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PREFACE 

A  serious  problem  confronting  program  planners  in  the  ' 
aerospace  industry  is  that  of  forecasting  the  probabilities 
of  success  and  failure  of  programs  during  the  initial  devel- 
opment stages.   Many  programs  have  final  cost  and  schedule 
overruns  amounting  to  more  than  double  the  initial  estimates, 
and  in  some  cases  these  overruns  necessitate  complete  can- 
cellation of  projects.   A  methodology  for  assessing  the 
technological  difficulty  of  advanced  programs  in  terms  of 
the  probability  of  success  within  stated  budgetary  and 
schedule  constraints  is  needed.   Such  a  procedure  would  per- 
mit early  evaluation  in  more  realistic  terms;  encourage  the 
setting  of  more  readily  attainable  goals  for  critical  pro- 
grams; and  permit  substantial  savings  in  national  resources 
by  singling  out  high-risk,  low-priority  projects  for  early 
cancellation. 

The  primary  purpose  of  this  study  is  the  derivation  of 
a  procedure  for  the  measurement  of  program  risk.   Past 
attempts  at  such  a  procedure  are  studied  and  evaluated.   The 
inputs  required  and  the  variables  of  greatest  usefulness  are 
selected  and  the  methodology  is  developed.   Suggestions  for 
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improvement  of  the  method  and  possible  areas  of  application 
are  then  treated.   Finally,  a  sample  program  is  used  to  ex- 
ercise the  model  developed  in  the  study  and  to  highlight  its 
strengths  and  weaknesses. 
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CHAPTER  I 
THE  MEASUREMENT  OF  RISK 

Introduction 

It  is  very  difficult  to  force  the  aerospace  industry  in 
the  United  States  into  the  generalized  models  used  in  eco- 
nomic analysis.   The  demand  is  monopsonistic ,  with  the  Federal 
Government  accounting  for  over  90  percent  of  the  purchases 
from  many  aerospace  firms;  however,  some  competition  does 
exist  among  the  various  branches  of  the  military  and  betv^een 
the  Department  of  Defense  and  the  National  Aeronautics  and 
Space  Administration. 

The  supply  side  presents  an  even  more  difficult  analy- 
tical problem.   Investigation  reveals  that  the  top  five  com- 
panies receive  less  than  one-fourth  of  the  total  dollar 
value  of  prime  contracts;  the  top  25  contractors  receive 
approximately  one-half;  and  the  top . 100  receive  less  than 
three-fourths  of  the  total  (1) .   These  figures  are  indica- 
tive of  the  large  number  of  firms  in  the  industry.   Entry  is 
restricted,  however,  by  the  extremely  large  capital  require- 
ments.  In  a  typical  program,  the  customer  will  issue  a  Re- 
quest for  Proposal  (PvFP)  or  a  Request  for  Quotation  (RFQ)  to 
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a   selected  group  of  contractors.  VJhile  it  is  possible  for  a 
contractor  who  vas  not  asked  to  bid  to  submit  an  unsolicited 
proposal,  the  normal  procedure  is  for  the  bidding  to  be  re- 
stricted to  those  firms  selected  b}'-  the  Department  of  De- 
fense (DOD) .- 

Among  these  few  firms  a  strong  competition  begins  which 
will  .-ventually  culminate  in  one  of  them  being  selected  as 
prime  system  contractor.   Since  only  one  contract  is  usually 
awarded  for  a  given  weapon  system,  the  competition  must  take 
place  before  the  order  is  placed.  Although  a  losing  bidder 
may  occasionally  become  a  subcontractor  to  the  prime  con- 
tractor for  a  subassembly  of  the  system,  success  lies  in 
winning  the  prime  contract. 

A  complicating  factor  is  the  prevalence  of  non- price 
competition.   For  eicample,  Boeing  submitted  a  significantly 
lower  final  bid  on  the  TFX  (F-lll)  program  than  did  General 
Dynamics;  but  the  latter  won  the  contract  on  the  basis  that 
Secretary  of  Defense  McNamara  considered  their  cost  figures 
"more  realistic."  This  points  up  still  another  problem 
area  -  the  fact  that  firms  will  deliberately  underquote 
costs  in  a  proposal  for  the  research  and  development  program 


'•"A  notable  exception  to  this  rule  is  MacDonald  Aircraft 
Corporation's  highly  successful  F-4  aircraft,  v;hich  resulted 
from  an  unsolicited  proposal. 
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and  purposely  take  a  loss  in  order  to  secure  the  business, 
safe  in  the  knowledge  that  these  losses  can  be  recouped  by- 
overstating  costs  in  the  production  program  which  will 
follow.*  Practices  such  as  these  imply  that  there  is  a 
large  element  of  risk  in  the  aerospace  industry.   Perhaps 
the  first  sort  of  risk  which  comes  to  mind  is  the  risk  that 
the  firm  will  not  secure  the  contract.   This  is  a  competi- 
tive risk  which  must  be  borne  by  every  firm  in  the  market- 
place economy  and  will  not  be  treated  in  this  discussion. 
The  questions  addressed  here  are  the  more  basic  ones  of 
whether  the  firm  can  afford  to  bid  at  all  on  a  program; 
after  the  decision  to  bid,  what  is  the  lowest  realistic 
figure  to  bid;  and,  after  the  contract  has  been  secured, 
what  are  t;he  chances  of  successful  completion  of  the  pro- 
gram. The  remainder  of  this  study  will  be  devoted  to  an 
investigation  of  these  aspects  of  aerospace  contracting. 

Types  of  Risk  in  Aerospace  Contracting 

The  most  familiar  treatment  of  the  subject  of  risk  is 
probably  Frank  H.  Knight's  Risk,  Uncertainty,  and  Profit  (2). 


*Similar  results  may  be  observed  in  some  cases,  even 
though  the  contractor  submits  his  bids  in  good  faith, 
through  the  operation  of  what  has  been  referred  to  as  an 
"optimistic  bias"  on  the  part  of  estimators.   This  phenome- 
non will , be  mentioned  again  in  Chapter  IV. 
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Knight  differentiates  betvjeen  risk,  vi'hich.  he  defines  as  an 
event  which  has  a  known  a  priori  probability  of  occurrence , 
and  uncertainty,  which  is  the  occurrence  of  total!}'  unpre- 
dictable events.   If  a  contingency  is  insurable,  it  is  risk; 
if  not,  it  is  uncertainty.   Knight's  thesis  is  that  since 
risk  is  predictable  it  is  not  a  rationale  for  (economic) 
profit  but  that  the  bearing  of  uncertainty  is  a  legitimate 
argument  for  profits.   In  the  aerospace  industry,  the 
government  assumes  all  or  a  portion  of  this  uncertainty  when 
it  negotiates  a  cost-plus- fixed- fee  or  a  cost-plus- incentive- 
fee  contract. 

There  are  other  types  of  "risk"  involved  in  the  opera- 
tion of  an  aerospace  firm.   The  stockholder  fee'J.s  that  he 
assumes  a  type  of  risk  by  purchasing  shares  of  the  company's 
stock.   The  firm  assumes  a  type  of  risk  when  it  invests  the 
stockholders'  funds  in  fixed  plant  and  equipment  or  in  "basic 
research  or  in  development  of  a  product  for  which  the  poten- 
tial customer  has  not  yet  issued  a  firm  contract.   Risk  is 
assumed  by  both  parties  when  a  contract  is  signed:   normally 
a  product  of  acceptable  quality  must  be  produced  by  a  dead- 
line and  V7ithin  prescr5.bed  cost  limits.   If  tlie  contractor 
does  not  deliver,  he  runs  the  risk  of  financial  loss  and 
loss  of  good  name  V7ith  the  customer;  the  customer  runs  the 
risk  of  being  unable  to  meet  the  military  requirements  of 
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the  nation  adequately  if  the  contractor  is  late  or  the 
quality  is  substandard, 

■    Another  concept,  attainment  risk,  is  the  uncertainty  of 
obtai'.ing  an  acceptable  production  result  in  terms  of  pre- 
state  1  specifications.'*   It  is  possible  to  break  attainment 
risk  ;'nto  several  component  risks.   Schedule  risk,  for  exam- 
ple, :  s  the  uncertainty  of  completing  a  project  before  a 
prescribed  deadline;  cost  risk  is  the  uncertainty  of  com- 
pleting a  project  V7ithin  predetermined  budgetary  restraints; 
performance  risk,  or  quality  risk,  reflects  the  uncertainty 
of  meeting  the  physical  performance  and  reliability  specifi- 
cations set  forth  in  the  contract.   The  recent  trend  toward 
incentive  contracts  represents  an  attempt  by  the  government 
to  reduce  attainment  risk  by  inducing  contractors  to  meet  or 
surpass  contractual  goals.   Hagen  (4)  concludes  that  fixed- 
fee  contracts  are  never  Pare to- optimal  and  can,  in  a  wide 
range  of  cases,  be  replaced  by  incentive- fee  contracts, 
V7hich  V7ill  increase  the  utility  of  both  parties. 

There  are  other  significant  forms  of  risk  that  can  be 
generally  classed  as  prestige  risks.   The  success  or  failure 


^General  Electric 's  Risk  Appraisal  of  Programs  System 
(RAPS)  defines  risk  in  this  v/ay:   "Risk  ...  can  be  defined 
as  the  probability  that  the  work  being  done  will  miss  the 
triple  target  of  cost,  delivery  schedule,  or  technical  per- 
formance." (3) 
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of  United  States  space  projects  is  a  determinant  of  public 
opinion  in  many  p^irts  of  the  v7orld.   Since  some  nations  rely 
heavily  on  the  purchase  of  American  military  machinery  for 
their  defensive  capabilities,  the  success  or  failure  of 
American  programs  can  have  international  ivaplications. 
Thus  J  the  failure  or  cancellation  of  a  major  project  could 
materially  affect  United  States  prestige  abroad. 

Prestige  risk  can  also  be  an  important  factor  V7ithin 
the  industry  since  a  firia's  reputation  can  be  a  significant 
asset  or  liability.   Sales  to  the  public  are  dependent  in 
part  on  company  prestige.   Even  if  the  company  sells  exclu- 
sively to  the  government,  prestige  or  reputation  may  be  the 
deciding  factor  in  contract  av.'ards.   The  "Department  of  De- 
fense Evaluation  of  the  Performance  of  Major  Contractors" 
(5)  is  a  publication  which  quantifies  company  prestige  in 
the  eyes  of  the  major  customer  of  the  aerospace  industry. 
This  rating  can  be  an  important  determinant  of  a  company's 
position  in  the  industry.   The  reputation  of  a  company  can 
also  affect  hiring  policies  and  other  non- sales  areas.   Pres- 
tige risk  can  therefore  be  a  significant  determinant  in  pro- 
gram evaluation. 

These  three  major  types  of  risk  are  highly  interrelated. 
The  failure  of  a  contractor  to  produce  a  product  of  accept- 
able quality  can  involve  economic  and  prestige  losses  to  the 
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company.   The  customer  can  also  incur  economic  and  prestige  ' 
losses  by  being  forced  to  pay  more  than  expected  for  the 
product  or  by  being  forced  to  release  an  inferior  system  to 
allied  countries.   The  allocation  of  risk  betv.-'cen  the  two 
parties  will  be  determined  primarily  by  the  type  of  con- 
tract.  The  Federal  Government  has  assumed  much  of  the  risk 
in  research  and  development  programs.   Pvecent  years  have 
seen  a  trend  toward  the  incentive  type  contract  for  programs 
V7hich  will  produce  a  physical  piece  of  hardware,  which  indi- 
cates an  implicit  recognition  of  the  risks  associated  with 
aerospace  programs  and  a  desire  to  share  the  risks  between 
the  contractor  and  the  customer.   Hagen  shows  that  the  in- 
centive contract  can  result  in  greater  utility  for  both  con- 
tractor and  customer  through  more  equitable  risk  distribu- 
tion. 

This  brief  discussion  suggests  that  the  ability  to 
measure  risk  v/ould  increase  the  ability  of  the  program  plan- 
ner to  choose  the  specific  contractual  terms  which  V70uld 
result  in  the  best  possible  allocation  of  the  risk  asso- 
ciated with  a  program.   In  the  sections  which  follow, 
specific  benefits  of  a  risk  measurement  will  be  suggested 
and  some  of  the  attempts  v.-hich  others  have  made  at  risk 
measurement  V7ill  be  reviewed. 
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The  Need  for  Quantitative  Risk  Measurement 

If 

After  considerable  study  and  discussion  with  experi- 
enced aerospace  planners,  it  was  decided  that  soms:  type  of 
attainment  risk  measure  would  be  the  best  possible  device 
for  evaluating  the  various  types  of  risk  which  are  associ- 
ated with  aerospace  programs.   It  could  explicitly  consider 
financial  attainment,  quality  or  performance  attainment,  and 
time  or  schedule  attainment;  it  implicitly  considers  pres- 
tige risk  by  assuming  that  it  is  a  function  of  those  types 
of  risk  which  are  considered  explicitly. 

A  risk  measure  which  could  assign  a  more  specific  risk 
classification  to  a  project  could  be  used  to  reevaluate  pro- 
gram cost  and  performance  in  more  than  general  terms.   A 
major  portion  of  the  $490  million  loss  sustained  by  General 
Dynamics  on  its  880-990  commercial  jet  program  could  have 
been  saved  if  adequate  assessment  of  the  budget,  schedule, 
and  performance  risks  of  the  program  had  been  made  at  the 
outset,  or  at  any  of  several  points  during  the  program.   For 
an  excellent  discussion  of  this  program  and  the  reasons  for 
the  losses,  see  "Hov/  a  Great  Corporation  Got  Out  of  Control" 
(6)  and  "GD :   The  Hard  Road  Back  from  the  Brink"  (7). 

Another  case  in  point  is  the  Skybolt  missile  program. 
VJhen  this  program  was  cancelled  in  1963,  the  program  costs 
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had  risen  from  1960  estimates  of  $893  million  to  almost  $3 
billion,  p.nd   the  program  V7as  far  behind  schedule  (8)  .   Over 
$150  million  was  considered  to  be  an  unsalvageable  loss  (9) . 
If  Skybolt  had  been  reevaluated  by  an  accurate  risk  measure 
v.-'hile  still  under  feasibility  analysis,  the  project  could 
have  been  altered  considerably.  The  congressional  contro- 
versy could  have  occurred  at  the  tirae  of  the  initial  appro- 
priation and  not  after  millions  of  dollars  had  been  v/asted. 

The  Skybolt  program  could  have  been  cancelled,  post- 
poned, or  accepted  under  revised  cost  and  schedule  esti- 
mates; if  the  project  were  accepted,  funding  levels  would 
have  been  more  realistic.   If  the  cancellation  had  come 
early  in  the  development,  a  considerable  amount  of  time  and 
money  would  have  been  saved;  and  Great  Britain,  v.liich  had 
planned  to  purchase  the  missile  system,  could  have  redefined 
its  iTiilitary  program  at  a  more  convenient  time.   This  is 
just  one  example  of  a  situation  where  failure  to  adjust  cost 
and  schedule  estimates  with  full  consideration  of  the  risk 
involved  led  eventually  to  cancellation  of  the  program. 

Both  government  and  industry  have  wasted  valuable  re- 
sources when  projects  were  cancelled  or  extensively  rede- 
fined late  in  the  program.   Consequently,  both  governm^ent 
and  industry  would  benefit  from  a  risk  measurement  which 
could  quantify  program  risks  during  the  early  phases  of 
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development.  Recent  years  have  seen  real  advances  in   the 
accuracy  of  estimating  techniques.   Unfortunately,  there  ha; 
been  an  even  greater  increase  in  the  requireip.ent  for  still 
more  sophisticated  techniqiies  and  better  estimating  proce- 
dures.  Better  estimating  methods  incorporating  risk  adjust- 
ing procedures  are  becoming  even  more  important  in  modern 
contracting  and  planning  in  the  aerospace  industry. 


Past  Attempts  at  Risk  Measurement 


Introduction 

Since  the  concept  of  risk  and  its  role  in  decision 
making  in  the  aerospace  industry  has  been  treated  before, 
this  section  will  be  a  survey  of  previous  studies  of  risk  in 
aerospace  programs.   An  extensive  search  of  the  current 
3.iterature  has  disclosed  very  little  documentation  of  con- 
cepts or  methodologies  for  the  generation  of  a  risk  index. 
Several  studies,  however,  have  considered  some  form  of  risk 
or  uncertainty.  Among  these  are  Alchian  (10) ;  Marshall  and 
Heckling  (11);  Peck  and  Scherer  (12);  Polski,  Clausen,  and 
Paige  (13)  ;  Summers  (14) ;  and  Terrell  (15)  .   The  purpose  of 
this  section  V7ill  be  to  relate  pertinent  conclusions  and 
similarities  among  the  above-mentioned  studies  to  appropri- 
ate areas  investigated  in  this  report. 
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.  In  1950,  A.   A.  Alchian  of  RAKD  Corporation  published  a 
study  of  the  variabilit}''  of  actual  costs  frora  cost  esti- 

A 

mates.   The  reliability  of  estiraates  froni  four  sources  V7as 
studied.  The  four  types  of  estimates  with  the  average  error 
for  the  trade  journal  data  studied  were  as  follows: 

(1)  Engineering  Estimates  ±25% 

(2)  Cost  Estimator's  Estimates  +23% 

(3)  Public  Engineers'  Construction  Cost  Estimators  ±167. 

(4)  Contractors'  Bid  Estimates  ±21% 

Alchian  concluded  that  the  difference  between  the  estimate 
and  the  realized  true  cost  for  an}'  device  of  an  improved 
quality  v7ould  usually  be  larger  than  any  of  the  above  devi- 
ations.  The  study  dealt  only  v'ith  the  unreliability  of  cost 
estimates,  and  no  technique  V7as  presented  to  obtain  better 
cost  estimates. 

The  most  extensive  analysis  of  risk  in  weapons  acquisi- 
tion V7as  made  by  Peck  and  Scherer.   They  observed  that  gen- 
erally weapons  program  decisions  involve  four  basic  elements: 
the  potential  enemy  threat,  cost,  the  state  of  the  art,  and 
time.   Uncertainties  in  estimating  the  last  three  items  - 
the  technical  feasibility,  the  development  time,  and  the 
development  and  production  costs  -  are  the  primary  con- 
tributing factors  of  program  risk. 
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Although  irf;ny  factors  were  considered  qualitatively,  no 

analytical  solutions  V7ere  derived  for  estiraating  any  aspects 

of  program  success.   Peck  and  Scherer  recognize  the  immense 

complexity  of  quantitative  risk  analysis  by  saying: 

The  existence  of  these  significant  uncer- 
tainties plai.nly  complicates  the  use  of  quanti- 
tative analysis  in  program  decisions.   Indeed, 
the  complications  from  uncertainty  are  so  great 
that  our  entire  approach  of  applying  an  optimi- 
zation model  to  the  program  decision  can  be 
questioned.* 

State  of  the  Art  Measurement 


A  single  facet  of  this  complexity  is  reflected  in  the 
technological  advance  inherent  in  a  given  program.   For  a 
comparative  analysis  of  programs  to  be  effective,  it  is 
essential  that  the  technological  complexities  be  compared. 

The  methodology  employed  by  Peck  and  Scherer  for  evalu- 
ating technological  difficulty  V7as  simply  a  v/ithin- sample 
ranking  of  paired  programs  (nine  programs  being  used)  by 
expert  opinion.  The  criterion  used  was  the  selection  of 
that  weapon  system  from,  each  pair  which  represented  the  more 
ambitious  state  of  the  art  advance  (SOA-A) ,  considering  the 
time  period  during  which  it  V7as  developed.  The  ranking  v^as 
then  arranged  on  a  scale  from  zero  to  one  hundred  units  to 


^'Reference  12,  page  303. 
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compare  the  relative  degrees  of  difficulty  or  technological 
advance.   This  methodology  (the  ranking  concept)  has  limited 
applicability  when  one  tries  to  measure  the  technological 
advance  of  a  new  program  which  is  not  functionally  compar- 
able to  any  member  of  the  sample.   The  main  problem,  is  de- 
termining a  realistic  technique  for  qtiantitative  measurement 
of  technological  advance. 

One  measure  of  program  success  often  used  is  the  rela- 
tive size  of  overruns*  in  budgets  and  schedules  encountered 
at  completion.   In  the  analysis  of  12  programs,  Peck  and 
Scherer  determined  an  average  cost  overrun  of  3.2^'^  and  a 
corresponding  development  time  overrun  factor  of  1.36. 
Table  1  shov7s  the  individual  results  for  the  12  programs. 

In  their  study  of  22  Air  Force  weapon  system  develop- 
ments, Marshall  and  Heckling  noticed  a  similar  trend  in  the 
production  cost  overruns.   This  trend  was  still  evident  even 
after  the  latest  cost  estimates  V7ere  deflated  by  a  price 
index  and  adjusted  for  output  quantity  changes.   Cargo  and 
tanker  aircraft  had  the  sm.allest  average,  1.2;  fighters  had 


^Overrun  is  the  ratio  of  final  actual  to  initial  esti- 
mate. 

^''^This  overrun  is  considerably  larger  than  the  one  found 
by  Alchian.   The  probable  reason  is  that  the  Peck  and 
Scherer  data  contained  only  large  programs,  while  som.e  of 
Alchian 's  data  were  from  smaller,  less  ambitious  projects. 
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Table  1 


DEVELOPMENT  COST  AND  TBIE  VARIANCE  FACTORS 
^  IN  12  WEAPONS  PROGRAMS''' 


Program 
Cost 

Developaient  ^ 
Cost  Factor^" 

Development 
Time  Factor" 

A 

4.0 

1.0 

B 

3.5 

2.3 

C 

5.0 

1.9 

D 

2.0 

n.a. 

E 

n.a. 

0.7 

F 

7.0 

1.8 

G 

3.0 

1.3 

H 

2.0 

1.0 

I 

2.4 

1.3 

J 

2.5 

1.3 

K 

0.7 

1.0 

L 

3.0 

1.4 

Average 

3.2 

1.36 

"■"'from  Peck  and  Scherer  (12),  page  429. 

^'Development  factors  are  ratios  of  final  actuals 
to  initial  esti.mates. 


an  overrun  of  1.7;  bombers,  3.0;  and  missiles,  5.2.   The  22 

programs  vjere  then  grouped  by  experts  according  to  degree  of 

technological  advance.   The  average  production  cost  overruns 

for  various  levels  of  advance  were  as  follo-,\7s: 

low  advance  1.35 
medium  advance  1.75 
high  advance      4.30 

This  trend  of  increasing  cost  overrun  with  increased 
technical  advance  supports  similar  findings  by  Peck  and 
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Scherer.  For  the  above  programs,  Marshall  and  Heckling  de- 
termined average  cost  and  schedule  overruns  of  3.01  and  1.28 
respectively.  In  the  development  of  military  weapons,  there 
is  a  tendency  to  place  more  emphasis  on  meeting  the  perform- 
ance capability  even  if  this  means  sacrificing  cost  and  time 
schedules.  It  is  noteworthy  that  in  most  cases  the  cost 
overrun  has  the  larger  ratio. 

Marshall  and  Heckling  do  not  present  a  m.ethodology  for 
predicting  the  final  outcome  of  a  program.   Hov7ever,  the 
above  results  indicate  that  the  probability  of  program  over- 
runs increases  directly  though  not  necessarily  proportion- 
ally V7ith  the  amount  of  technological  advance  required  by 
the  progrc'm. 

Peck  and  Scherer  conclude  that  the  unpredictability  of 
a  weapon  system  program  manifests  itself  primarily  in  the 
cost  dimension  due  to  the  premium  placed  on  performance  and 
the  tradeoffs  among  performance,  time,  and  resources. 


Quantitative  Risk  Assessment 


It  becomes  readily  apparent  that  there  exist  many 
forces  which  affect  the  outcome  of  a  program,  some  of  which 
interact  in  a  complex  array  of  tradeoffs,  as  in  the  case  of 
scheduling  and  funding.   There  are  also  the  parameters  which 
are  physically  unmeasurable  but  still  qualitatively  compared 
by  expert  opinion  in  the  appropriate  fields. 
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The  first  well-known  analytical  study  involving  uncer- 
tainty was  performed  by  Summers.   This  was  a  statistical 
study  to  adjust  original  cost  estimates  by  an  analytic  func- 
tion such  that  the  revised  value  would  be  closer  to  the  actual 
outcome.   Based  on  22  weapon  systems  and  68  cost  estimates, 
Summers'  study  found  these  three  variables  to  be  most  impor- 
tant : 

1.  time  the  estimate  is  made  in  relation  to  the 
development  program 

2.  degree  of  technological  advance  required 

3.  length  of  the  development  period. 

The  result  of  Summers'  analysis  is  an  exponential  function, 

In  F  =  a^  +  a^t  +  a2At  +  a3A  -h  a^k'^  +   a3L  +  a^T  +  u,-v 

which  can  also  be  expressed  in  the  equivalent  form, 
F  =  Ke^lt  ^a2tA  ^asA  ga4A2  ^35!  ^361  ^^.. 

where  the  following  parameters  are  defined: 

K,a  =  constants, 

t  =  a  fraction  representing  the  time  elapsed  in 
the  program  when  the  estimate  was  made, 

A  =  a  numerical  measure  representing  the  techno- 
logical advance  in  the  program, 

L  =  the  length  of  the  development  period, 

T  =  the  calendar  year,  and 

u,v  =  error  terms. 


^Reference  14,  page  33, 
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Once  again  the  problem  of  measuring  the  inherent  tech- 
nological advance  of  the  program  arises.   The  technique  used 
in  evaluating  A  was  a  vjithin- sample  ranking  determined  by 
RAND  engineers.   For  reasons  given  previously,  difficulties 
occur  in  consistently  applying  this  measurem.ent  scheme  to 
new  programs . 

In  estimating  techniques  that  utilize  data  of  past  pro- 
grams. Summers  concludes 

...it  must  be  emphasized  that  the  crudeness 
of  the  data  precluded  the  possibility  of  obtain- 
ing anything  like  a  precise  description  of  cost 
estimating  errors  in  the  years  following  VJorld 
War  II...,  What  is  claimed  is  that  the  methods 
employed  in  this  study  constitute  a  reasonable 
V7ay  of  looking  hard  and  close  at  the  information 
available.   The  merit  of  the  formulation  offered 
here  lies  in  its  explicit  method  of  derivation 
and  in  the  possibility  it  offers  for  future  re- 
finement V7ith  miore  complete  data  and  V7ith  experi- 
ence of  new  kinds  of  systems  and  new  production 
techniques.'' 

To  recapitulate,  the  results  of  Summers'  study  are-  simi- 
lar to  those  of  other  investigations  reported  in  this  sec- 
tion.  These  results  may  be  summarized  as  follov7s : 

•  The  use  of  a  subjective  ranking  by  experienced 
engineers  to  evaluate  the  state  of  the  art  ad- 
vance of  the  program. 

•  The  conclusion  that  the  average  cost  overrun  in- 
creases with  technological  difficulty  and  de- 
creases with  respect  to  how  close  the  program  is 
to  completion  when  the  estimate  is  made. 


''Reference  14,  page  11. 
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In  1964,  at  the  AIAA  Annual  Meeting,  Polski,  Clausen, 
and  Paige  presented  a  clear  and  well-defined  methodology  for 
reducing  risk  on  research  and  development  (R&D)  programs. 
The  purpose  of  such  a  methodology  is  as  follov7s: 

Such  a  system  has  benefits  for  both  custoraers 
and  contractors  because  an  increase  in  knowledge 
of  tVie  risks  leads  to  more  sound  decisions  con- 
cerning them  and  general  improvements  in  program 
management  plans,  methods,  and  communications. 
Greater  knowledge  of  risks  may  also  lead  to  ulti- 
mate improvements  in  such  areas  as: 

•  Source  Selection 

•  Program  Definition 

•  R&D  Program  Value" 

Four  ranges  of  risk  -  (1)  high,  (2)  moderate,  (3)  minor, 
and  (4)  low  -  are  used  in  the  analysis.   A  risk-end  iteui 
matrix  is  generated  along  with  suggestions  and  costs  for 
lowering  the  risk  in  the  defined  areas.   (See  Figure  1.) 
This  is  the  first  V7ell- defined  methodology  for  utilizing  the 
knowledge  of  a  wide  variety  of  people  involved  in  a  program. 
This  iiTiplies  that  a  risk-cost  tradeoff  matrix  could  be  found. 

Terrell  attempted  to  unify  the  areas  that  contributed 
most  to  uncertainty  v/ith  a  more  quantitative  assessment  of 
risk  for  programs  in  the  early  phases  of  definition.   A 
methodology  is  presented  which  quantitatively  compares  the 
technological  advance  of  different  programs.  This 


"'Reference  13,  page  1. 
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measurement  begins  as  the  ratio  of  the  proposed  development 
item  to  the  best  available  (developed)  item  and  is  then  ad- 
justed by  use  of  a  theoretical  limit  assessment  curve.   This 
curve  takes  into  consideration  the  relationship  of  the  de- 
sired performance  capability  to  the  theoretically  possible  • 
performance  levels.  A  procedure  is  developed  for  applying 
standard  statistical  techniques  to  the  performance  character- 
istics for  more  than  ten  programs  in  order  to  generate  a 
family  of  attainment  curves.  These  curves  predict  the  prob- 
ability of  attaining  performance,  reliability,  and  schedule 
with  corresponding  tradeoffs  involving  cost.   (See  Figure  2.) 

The  projects  included  in  Terrell's  sample  showed  an 
average  cost  overrun  of  2,5  and  a  schedule  overrun  of  1.2. 
These  values  fall  into  the  general  ranges  of  values  deter- 
mined by  previous  researchers. 

In  conclusion,  with  the  attainm.ent  prediction  equations, 
a  specific  technique  for  evaluating  the  degree  of  difficulty 
in  a  program,  and  considerations  involving  the  theoretical 
limit,  Terrell's  study  is  the  most  advanced  attempt  at  the 
evaluation  of  a  risk  index. 

Summary 

This  chapter  indicates  something  of  the  variety  of  con- 
cepts which  have  been  analyzed  under  the  name  "risk."  The 
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need  for  a  quantitative  procedure  for  the  measurement  of 
attainment  risk  is  established,  and  a  survey  of  the  v^ork  to 
date  is  presented. 

A  summary  of  the  contributions  of  the  principal  studies 
in  th J  area  of  risk  is  given  in  Table  2.   The  evolutionary 
chara  -ter  of  the  development  and  the  sources  of  some  of  the 
conce"  ts  presented  in  this  study  are  indicated.   Included 
are  the  primary  areas  of  interest,  functional  relationships 
developed,  and  a  brief  summary  of  the  important  conclusions 
of  each  study. 
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CHAPTER  II 
STATE  OF  THE  ART  ADVANCE  (SOA-A)  MEASOREMENT 

Introduction 


In  almost  all  of  the  work  which  has  been  done  on  the 
problem  of  risk,*  the  researchers  have  concluded  that  the 
most  important  single  parameter  in  the  assessment  of  Risk  is 
the  advancement  in  the  so-called  state  of  the  art,  or  the 
increase  in  the  level  of  technology  which  is  required  to 
accomplish  the  project  under  consideration.   Since  this 
study  deals  primarily  with  research  and  development  pro- 
gram.s,  it  is  intuitively  evident  that  projects  which  require 
greater  technological  advances  will  generally  involve 
greater  risks.   Therefore,  if  a  Risk  index  is  to  be  devel- 
oped, one  should  first  address  the  problem  of  deriving  a 
methodology  for  the  assessment  of  the  state  of  the  art  ad- 
vance (SOA-A)  associated  with  a  project.   It  is  important 
that  the  system  developed  be  logically  consistent  and  that 


^Throughout  the  development  which  follows,  the  capital- 
ized "Risk"  will  refer  to  the  specific  concept  of  attainment 
risk  as  discussed  in  the  first  chapter  of  this  study,  with 
the  low^r  case  "risk"  referring  to  the  more  general  concept. 
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it  be  suf fic5.ently  general  to  be  applicable  to  a  wide  range 
of  program  types  V7hile  at  the  same  time  requiring  a  common 
methodology  and  thereby  assuring  a  measure  of  consistency  in 
SOA-A  measurements  made  by  different  analysts  on  different 
projects . 

Peck  and  Scherer  indicate  that  what  they  call  "State  of 
the  Art  Index"  measures  "the  technical  achievement  and  inno- 
vation required  to  accomplish  a  quality  (performance  and 
reliability)  objective.""  They  place  a  project  on  a  scale 
which  runs  from  zero  to  one  hundred  and  assign  it  a  number 
relative  to  the  other  (historical)  projects  already  placed 
upon  the  sawe  scale.   This  approach  is  based  upon  the  ex- 
perience and  judgment  of  those  in  the  field  and  may  be  re- 
ferred to  as  a  completely  intuitive  approach.   An  alternate 
approach  would  be  one  in  V7hich  only  quantitative  factors  are 
considered;  experience,  judgment,  and  intuition  would  not 
play  an  im.portant  role  in  the  synthesis  of  a  value  for  the 
SOA-A  of  a  project.   In  the  first  place,  sufficient  data 
are  not  available  to  permit  this  type  of  analysis;  in  the 
second,  even  with  perfect  data  it  v7ould  still  possess  cer- 
tain shortcomings  since  V7hat  this  type  of  analysis  gains  in 
objectivity,  it  forfeits  in  the  loss  of  the  invaluable 
qualities  of  experience  and  judgment. 


"'Reference  12,  page  289. 
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While  the  approach  presented  here  will  not  be  com- 
pletely objective,  in  that  it  makes  some  use  of  subjective 
or  intuitive  judgments,  it  attempts  to  force  the  analyst  to 
quantify  at  a  lov/er  level  and  to  follow  a  set  procedure  for 
determining  the  SOA-A  of  a  project.   This  may  be  called  a 
subjective-analytic  approach . 


Development 


Given  an  advanced  project  proposal,  the  most  important 
performance  criteria  (e.g.,  speed,  weight,  accuracy,  thrust, 
and  range)  are  listed  in  order  of  decreasing  importance 
(xi...Xj^),  and  each  criterion  is  given  a  v/eight  which  indi- 
cates its  importance  to  the  success  of  the  program  relative 
to  the  other  criteria  (ai...aj^).   Each  successive  weight  is 
less  than  or  equal  to  the  previous  one,  and  all  are  relative 
to  the  most  important  criterion  (xj^)  . 

«1  =^  1'  -i-1  -^i-  ^i-1-1 

Factors  which  should  be  analyzed  in  the  evaluation  of 
the  aj^  include  expected  development  cost,  expected  develop- 
ment time,  and  importance  to  the  success  of  the  project. 

Normally,  a  subsystem  which  is  expected  to  account  for 
50  percent  of  the  total  project  cost  will  be  m.ore  heavily 
weighted  in  determining  a  Risk  measure  than  will  one  vjhich 
is  estimated  at  10  percent  of  the  total.   Similar  reasoning 
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leads  to  the  same  conclusion  about  articles  v.'ith  different 
expected  development  times.   The  third  factor  depends  upon 
the  relative  value  the  customer  places  upon  the  various  fea- 
tures of  the  project.  There  are  a  few  major  design  features 
v;hich  are  of  extreme  importance  in  a  project.   These  are  the 
reasons  the  project  was  deemed  necessary  in  the  first  place „ 
In  addition,  there  vjill  be  other  correlative  or  subordinate 
"extra"  features  of  the  "nice  to  have"  variety.   In  a 
squeeze  to  produce  an  acceptable  product  by  a  deadline, 
these  secondary  measures  may  be  compromised  much  more 
readily  than  the  primary  design  features.   The  experienced 
analyst  can  pinpoint  these  distinctions  and  use  this  infor- 
mation to  assist  in  the  ranking  of  criteria  and  assigning  of 
the  aj^. 

The  criteria  having  been  selected  and  v/eighted  relative 
to  each  other,  the  next  step  is  to  determine  the  SOA~A  rep- 
resented by  each  of  the  criteria  on  the  list.   Let  the  SOA~A 
for  X£  be  designated  by  Sj^,  where  Sj^  is  defined  to  be  the 
ratio  of  the  desired  performance  level  to  the  performance 
capability  of  the  most  advanced  currently  available  (devel- 
oped) model. 


Sa    = 


desired  performance  level 


^   current  demonstrated  capability 
In  cases  where  an  increase  in  performance  results  in  a 


-  28  - 

decrease  (e.g.,  weight,  volume,  resolution),  simply  invert 
the  ratio  to  obtain  S., 

At  this  point  in  the  development,  a  set  of  numbers  has 
been  computed  that  indicates  the  relative  magnitudes  of  the 
performance  increases  that  will  be  required  by  the  proposed 
system.   Now  it  becomes  desirable  to  adjust  these  perform- 
ance ratios  by  taking  theoretical  limits  into  consideration. 
As  a  technological  breakthrough  occurs,  new  performance 
capabilities  are  realized;  but  a  given  level  of  technologi- 
cal capability  can  be  exploited  only  so  far.   The  principle 
of  diminishing  marginal  returns  begins  to  operate;  and  in- 
creasingly greater  amounts  of  time,  money,  and  other  re- 
sources are  required  for  additional  performance  advances.* 
The  models-accounts  for  this  fact  by  adjusting  the  S^  for 
theoretical  limit  considerations.   An  adjusted  value  for  S^ 
(S.)  can  be  obtained  by  referring  to  the  theoretical  limit 
curve  of  Figure  3.    The  curve  is  used  by  estimating  the 


*At  the  13th  annual  meeting  of  the  American  Astronauti- 
cal  Society,  Edv/ard  N.  Hall  presented  a  paper  on  An  Econom- 
ical Approach  to  Space  Transportation  (16)  ,  in  which  he 
demonstrates  an  exponential  relationship  between  the  ratio 
of  actual  final  program  cost  to  initial  estimates  of  program 
cost  and  the  change  (or  advance)  in  the  SOA  which  is  sought. 
This  supports  the  author's  choice  of  the  exponential  form 
for  the  T  curve  of  Figure  3. 

^'^The  curve  in  Figure  3  was  postulated  by  the  author  af- 
ter analyzing  Terrell's  curve  and  discussing  the  subject  at 
great  length  with  experienced  aerospace  planners.   It  should 
be  determined  empirically  as  more  data  become  available. 
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degree  to  which  the  present  level  of  technological  development 
has  been  exploited  and  the  approximate  percent  of  the  theore- 
tical limit  which  has  been  attained.   Entering  this  value  on 
the  abscissa,  the  corresponding  value  of  T^  can  be  found  oh 
the  ordinate.  IT  is  a  dimensionless  number  which  indicates  the 
technological  difficulty  of  the  project.   This  T curve  is 
determined  empirically  by  assigning  S^   values  to  criteria  in 
historical  programs  and  determining  the  corresponding  values 
for  the  V-.      The  curve  may  be  revised  from  time  to  time.   It 
may  also  be  necessary  to  define  different  curves  for  different 
classes  of  programs. 

The  value  of  S.  is  obtained  by  multiplying  S^  by  T^. 

I 

Thus,  a  set  of  values  for  the  S.  has  been  established  which 

takes  theoretical  limit  considerations  into  account.   Finally, 

t 

the  individual  S^  measures  are  combined  in  a  weighted  summa- 
tion in  order  to  arrive  at  a  measure  of  the  SOA-A  of  the  total 
system  (SJ)  .  Mathematically: 

i=l 
Obviously,  the  series  S-j.  will  continue  to  increase  as 
the  value  of  n  increases.   It  therefore  becomes  necessary  to 
demonstrate  that  in  practice  S^  will  tend  to  approach  a  limit- 
ing value.   Study  of  past  programs  reveals  that  values  for  S^ 
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tend  to  lie  between  one  and  five;  and,  in  most  areas,  a  value 

of  three  would  indicate  a  large  increase.*  It  is  difficult 

to  imagine  an  aircraft  project  v/hich  would  require  an  increase 

in  speed  of  more  than  twice  the  present  capability  or  one 

which  would  require  a  threefold  increase  in  engine  thrust. 

Since  S.  is  the  product  of  S.  and  T- ,  its  value  can  be 
1        *^  11' 

made  large  by  inserting  a  large  value  for  either  S.  or  Tj^. 
If  T.  is  large,  the  technological  difficulty  is  great;  and 
the  relative  amounts  of  time  and  dollars  required  will  norm- 
ally be  large;  therefore,  one  should  expect  a  high  value  for 
a..   Since  large  values  for  T.  indicate  reduced  cost-effective- 
ness, it  would  not  be  expected  that  unimportant  criteria  would 
be  designed  to  require  values  approaching  the  theoretical 
limit.   On.- the  other  hand,  it  is  possible  to  have  large  values 
of  S-  accompanied  by  small  values  of T-  (for  example,  before 
a  technological  breakthrough  has  been  exploited) .   Any 
criterion  which  requires  a  large  performance  increase  should 
rank  high  in  importance  to  the  project  and  will  therefore 
carry  a  relatively  high  a. . 


*Notable  exceptions  to  this  idea  are  found  in  the 
fields  of  electronics  and  computers.   Recent  advances,  such 
as  integrated  circuits,  have  given  rise  to  quantum  increases 
in  speed  and  decreases  in  weight  and  volume.   The  assumed 
relationships  among  high  values  for  S-i_,T'i,  S^,    and  a-  tend 
to  minimize  the  existence  of  these  exceptions  as  an  objection 
to  the  hypothesis  that  Sy   approaches  a  limiting  value. 
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As  indicated  above,  if  either  S^  or  V ■    is  large,  the 
product  will  tend  to  be  large  also.   Since  large  values  of 

either  S.  orT.  tend  to  be  associated  with  large  a^-  values, 

11  ^ 

it  may  be  concluded  that  S.  will  be  positively  correlated 
with  a..   Following  this  line  of  thought,  then  as  i  increases. 

t 

a.,  either T.  or  S.,  and  therefore  S^  will  decrease,  yielding 
a  relationship  between  Sm  and  n  which  has  the  form  of  Figure 
A,  where  SZ   is  the  hypothesized  limiting  value  of  S„.   Analy- 
sis of  over  twenty  real  and  hypothetical  programs,  ranging 
from  the  very  simple  to  the  extremely  ambitious  in  techno- 
logical complexity,  indicates  that  the  expected  range  of 


values  for  S,^  is 


0  i  S*  ^  1.2   . 


In  g^eral,  it  is  the  intention  of  the  author  that 
the  value  of  n  should  be  less  than  ten.   Beyond  this  value, 
the  relationships  among  the  a.  are  virtually  impossible  to 
assess  with  any  degree  of  accuracy.   The  procedure  is  designad 
to  permit  comparisons  among  similar  programs,  and  it  is  the 
intention  of  the  author  that  only  the  major  performance 
measures  be  included  in  the  list.   It  is  expected  that  n  will 
tend  to  have  a  value  of  four  to  six  in  most  instances.   Since 
the  methodology  is  applicable  to  a  wide  range  of  sorts  and 
sizes  of  projects,  it  can  be  used  to  evaluate  a  complete 
system,  then  be  reapplied  to  the  more  important  subsystems. 
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0 

Summary 

A  review  of  the  methodology  for  generating  the  state 
of  the  art  advance  (SOA-A)  for  a  project  would  be  instructive 
at  this  point.   The  steps  to  be  followed  (see  Figure  5)  are 
these: 

(1)  Select  the  performance  characteristics  which 
are  most  important  to  the  system  and  rank 
these  from  most  to  least  important. 

(2)  Assign  numbers  (a.)  to  each  of  these  criteria, 
indicating  their  relative  importance  to  the 

overall  project,  a-,    =   1;  ^±-i^   ^i~  ^i+1 

(3)  Calculate  the  S.  as  the  ratio  of  expected 
performance  to  the  best  currently  available 
model  performance,  inverting  the  ratio  in 
cases  where  improved  performance  results  in 
a  small  numerical  value  for  the  index. 

(4)  Estimate  the  percent  of  the  theoretical 
limit  which  the  desired  performance  ap- 
proaches.  Using  the  Theoretical  Limit  Curve, 
find  the  appropriate  values  for  T. . 

(5)  Adjust  the  S.  to  account  for  theoretical  limit 
considerations.   S.  =  ^j^S. 
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(6)   Combine  the  S^  in  a  weighted  summation  to  obtain 
a  measure  of  the  overall  SOA-A  for  the  total 
project. 

S  .  l       a,  s! 

1  =  1 

This  procedure  permits  the  use  of  subjective  judgment 
on  the  part  of  the  analyst  in  assigning  the  weights  to  the 
criteria  and  in  the  selection  of  the  criteria  themselves. 
It  requires  quantification  at  a  level  below  the  total  system 
level,  however,  and  thus  forces  the  analyst  to  examine 
critically  the  actual  makeup  of  the  final  composite  SOA-A 
measure.   This  procedure  combines  the  stronger  features  of 
both  the  completely  intuitive  approach  and  the  wholly 
analytical  method,  resulting  in  a  balanced  analysis,  which 
will  tend  to  give  more  consistent  SOA-A  measurements  in 
the  long  run. 


CHAPTER  III 
A  RISK  INDEX  METHODOLOGY 

Selection  of  Parameters 

SOA-At  (S^) 

In  the  preceding  chapter,  it  was  pointed  out  that  the 
state  of  the  art  advance  required  by  a  project  is  the  most 
important  single  factor  in  the  determination  of  an  overall 
Risk  index.   All  other  factors  held  constant,  the  Risk 
associated  with  a  program  would  increase  if  the  SOA-A  in- 
creases. ■<^In  the  absence  of  any  information  about  funding 
and  time  schedules,  the  SOA-A  would  be  a  reasonable  measure 
of  the  inherent  Risk  of  a  program.   This  will  be  one  of  the 
factors  used  to  develop  the  Risk  index. 

Performance 

Performance  attainment  is  a  vital  measure  of  program 
success.   Detailed  consideration  was  given  this  variable 
during  the  course  of  this  study.   In  the  final  analysis,  it 
was  decided  that  performance  would  not  be  used  explicitly  in 
the  procedure.   The  reasons  for  this  decision  were  these: 
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(1)  S^  measures  the  inherent  technological  difficulty 
of  meeting  performance  goals.   To  also  include  a 
measure  of  the  probability  of  meeting  these  goals 
would  be  a  form  of  double  counting. 

(2)  Historical  data  refer  primarily  to  programs  which 
have  met  (or  substantially  completed)  the  stated 
performance  goals . 

(3)  A  program  is  not  considered  to  be  finished  until 
the  performance  criteria  have  been  met  to  the  satis- 
faction of  the  procurer. 

(4)  \7hile  there  are  some  instances  in  which  projects 
have  been  declared  to  be  completed  when  not  all  of 
the  performance  goals  were  met,  these  are  the 
exceptions,  and  the  programs  were  in  all  likeli- 
hood not  considered  successful  by  either  contractor 
or  purchasing  agency. 

(5)  If  a  project  is  completed  before  the  time  and  cost 
constraints  have  been  encountered,  the  contractor 
would  tend  to  add  additional  capability,  but  only 
to  the  point  of  using  up  the  remaining  resources. 
Since  many  projects  overrun  these  resource  contraints, 
it  seems  improbable  that  contractors  would  volun- 
tarily exceed  the  required  performance  goals,  and 
that  the  overruns  would  be  small. 
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(6)   There  does  not  appear  to  be  a  systematic  method 
for  measuring  the  cost  and  time  overruns  for  a 
program  which  is  cancelled. 

In  light  of  these  facts,  it  was  decided  to  consider  the 
meeting  of  the  performance  goals  (or  substantial  compliance) 
to  be  a  constant  factor  at  100  percent  of  the  contract  spec- 
ifications and  that  this  would  be  a  benchmark  from  which  the 
other  variables  would  be  measured.   In  other  words,  the  total 
program  cost  (actual)  is  the  cost  up  to  the  point  when  a 
satisfactory  product  has  been  delivered  to  the  customer, 
and  this  date  will  mark  the  completion  of  the  project  in 
time,  as  well. 

Program  Cost 

Program  cost  is  a  tangible,  measurable  resource  con- 
straint in  any  program.   It  is  a  readily  available  item  of 
data  on  historical  programs  and  is  an  objectively  measurable 
quantity.   Virtually  every  attempt  at  risk  measurement  has 
included  cost  as  a  factor.  Any  discussion  of  Risk  must 
ultimately  be  expressed  in  terms  of  dollars  in  order  to  be 
useful  to  the  decision  maker,  since  any  discussion  of  Risk 
by  managers,  program  planners,  or  legislators  will  have 
economic  overtones.   Therefore,  cost  is  one  of  the  variables 
which  will  be  used  to  determine  the  Risk  index. 
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Time 

To  Government  planners,  considerations  of  time  are 
second  in  importance  only  to  those  of  cost.   To  the  military 
commander,  time  is  the  overriding  consideration,  since  the  • 
ultimate  x^eapon  delivered  a  day  (or  an  hour  I)  too  late  can 
be  totally  useless.   In  planning  for  the  defense  of  the 
nation,  the  meeting  of  delivery  schedules  is  vital.   Since 
the  1950 's,  this  country  has  been  engaged  in  a  vigorous 
competition  to  overcome  and  eliminate  various  "gaps"  (missile 
gap,  booster  gap)  and  to  out-perform  the  Soviets  in  other 
ways  (the  "Moon  Race") .   In  each  instance,  time  was  the  next 
most  important  criterion  after  performance  and  reliability, 
and  cost  goals  were  repeatedly  revised  upward  in  order  to 
meet  and  surpass  time  objectives.   Time  has  therefore  been 
chosen  as  the  third  variable  which  will  be  utilized  in  the 
derivation  of  an  index  of  Risk. 

Other  Considerations 

As  has  been  indicated  above,  some  previous  studies  have 
used  measures  other  than  the  three  selected  here.   Since  the 
approach  used  here  will  be  to  build  a  set  of  logical  rela- 
tionships, the  three  variables  which  best  describe  the  pro- 
gram have  been  chosen.   In  many  of  the  previous  studies  cited, 
the  approach  used  was  to  fit  hypothesized  functional  rela- 
tionships to  historical  data  by  use  of  a  least  squares 
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routine.   While  this  approach  gives  the  best  estimates  over 
the  sample,  it  does  not,  in  general,  give  the  best  estimates 
for  programs  not  included  in  the  sample.   Since  this  study 
attempts  to  develop  a  system  based  on  logic  rather  than 
empirical  fit,  these  three  basic  factors  were  selected. 

It  should  be  emphasized  that  almost  all  of  the  prior 
studies  relied  upon  these  three  as  inputs,  even  though  in 
many  instances  other  measures  were  added  to  give  a  better 
empirical  fit  to  the  data.   In  the  first  issue  of  Aerospace 
Management,  General  Electric  Company's  Missile  and  Space 
Division  presents  a  summary  of  their  Program  Appraisal  and 
Revision  (PAR)  system  which  lists  the  most  important 
variables  as  "Technical  Performance,  Schedule  Performance, 
and  Cost  Performance" (17) . 

Reasons  for  Subjective  Assessment 

The  chapter  of  this  report  dealing  with  the  method 
for  determining  the  SOA-A  of  a  project  indicated  the 
principal  advantages  realized  by  a  system  which  permits  the 
experienced  analyst  to  make  use  of  his  training  and  to 
bring  his  expert  judgement  to  bear  upon  the  problem.   It 
was  indicated  there  that,  while  data  collection  for  a  purely 
mechanical  system  would  be  a  virtually  impossible  task, 
even  the  availability  of  perfect  data  would  not  make  such 
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an  approach  attractive  since  it  precludes  the  use  of  the  in- 
valuable qualities  of  experience  and  judgment  on  the  part  of 
the  senior  analyst.   Consequently,  the  development  of  the 
Risk  index  begins  with  the  generation  of  two  sets  of  estimates 
one  for  program  cost,  the  other  for  development  time.   It  is 
assumed  here  that  these  estimates  are  derived  using  the  best 
currently  available  cost  and  schedule  estimating  procedures, 
and  that  they  are  consistent  with  the  other  estimates  which 
would  normally  be  made  for  the  program. 

Risk  Index  Procedure 

The  development  of  the  index  is  begun  in  a  manner  similar 
to  the  PERT  methodology.   Given  the  perform.ance  requirements 
of  a  program,  a  set  of  three  estimates  is  prepared  for  each 
of  the  input  variables  (dollars  and  time)  indicating  the  ex- 
pected values  of  resources  required  to  meet  the  performance 

goals:  * 

(1)  Low  -  an  optimistic  value  which  has  a  probability 
of  only  1  percent  of  actually  being  met,  even  if 
all  goes  well  and  no  major  delays  or  cost  increases 
occur. 

(2)  High  -  a  pessimistic  value  which  has  only  a  1 
percent  chance  of  not  being  met. 

(3)  Modal  -  an  estimate  of  the  most  likely  value,  taking 
into  consideration  past  projects  of  a  similar 
nature. 
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Using  these  three  estimates,  probability  (of  program 
success)  distributions  can  be  constructed  for  development 
time  and  cost.   Following  the  PERT  analogy,  the  Beta 
distribution  was  selected."   Figure  6  and  Figure  7  are  re- 
presentative of  possible  forms  for  such  distributions.   The 
preponderance  of  opinion  seems  to  indicate  that  the  right- 
skex^/ed  form  of  Figure  7  is  the  more  likely  one  for  both 
schedule  and  funding;  the  possibility  does  exist,  however, 
that  some  projects  may  be  better  described  by  the  skewed 
left  or  even  the  symmetrical  distribution.   This  possibility 
is  one  of  the  reasons  for  the  selection  of  the  versatile 
Beta  distribution. 

Superimposed  on  these  distributions,  the  actual  values 

of  the  contractual  (or  tentative)  values  for  time  and  funding 

(t   and  $  )  are  plotted.   The  shaded  areas  under  the  curves 
o      o 

to  the  left  of  these  resource  restrictions  represent  estimates 
of  the  probabilities  of  development  'time  and  cost  success  for 
the  program  ( ]^j-  and  |^g)  ;  that  is,  the  probabilities  that  the 
project  will  be  completed  within  the  assumed  budgetary  and 
schedule  restraints. 


*For  a  thorough  discussion  of  the  PERT  assumptions  and 
their  implications,  see  Ko  R.  McCrimraon  and  C.  A.  Ryavec, 
An  Analytical  Study  of  the  PEPvT  Assumptions  (18)  . 
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In  most  cases  the  modal  estimate  will  be  either  the 
best  available  estimate  or  the  contractual  figure,  since  the 
person  performing  the  analysis  will  base  his  estimates  upon 
information  from  the  group  v/hich  generated  the  contractual 
figures o   Even  though  the  value  for  $^   or  t  may  fall  on  or 
near  the  modal  value  of  the  frequency  distribution,  the 
shaded  area  (the  value  of  }l/^  ox  ]l/ ,^)   v/ill  be  a  functrion  of 
the  distribution  form,  and  vrill  therefore  differ  from  one 
distribution  (and  project)  to  the  next.   It  is  often  of 
interest  to  determine  the  effect  V7hich  a  change  in  funding 
or  time  from  the  expected  values  vjill  have  upon  the  proba- 
bility of  program  success.  As  will  be  shoxv^n  presently,  a 
sensitivity  analysis  permits  calculations  of  this  sort. 

If  a  sensitivity  anal^'^sis  is  to  be  attempted  or  if  there 

are  several  alternative  values  for  $  or  t  ,  the  generation 

o     o 

of  a  cumulative  distribution  function  from  the  frequency 
distribution  (see  Figure  8)  permits  a  much  faster  analysis 
while  at  the  same  time  making  the  model  more  easily  under- 
stood.  From  such  a  function  the  value  of  W^   or  V/^   can  be 
read  directly  from  the  ordinate  for  any  value  of  $   or  t^. 

The  probability  of  two  independent  events  both  occuring 
is  the  product  of  the  probabilities  that  each  will  occur. 

P(both  A  and  B)  =  P(A)  •  F(B) 
While  the  events  considered  here  (schedule  success  and  budget 
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success)  are  clearly  not  independent,  the  present  state  of 
development  in  Risk  analysis  is  sufficiently  crude  to  permit 
this  simplifying  assumption  in  the  analysis.   Among  others, 
Terrell  made  a  similar  assumption  with  reasonable  success. 

Relying  upon  past  empirical  evidence  as  a  partial  justi- 
fication for  the  simplifying  assumption,  the  product  of  the 
probability  of  cost  success  (]^$)  and  the  probability  of  time 
success  (  ]^^)  shall  be  used  to  represent  the  unadjusted  prob- 
ability of  success  for  the  total  program  (V^-n)  * 

The  unadjusted  probability  of  failure  for  the  project  (^  yd 
is  the  complement  of  )^jj. 

It  was  indicated  in  Chapter  II  that  the  value  of  the 
SOA-A  measure  (S„)  would  normally  fall  into  the  range 
0  ^  Srp  ^  1.2   .   Since  it  is  intended  that  the  estimate  of 
the  experts  shall  be  the  base  figure  which  is  adjusted  for 
SOA-A,  the  unadjusted  probability  of  failure  will  be  multiplied 
by  1  +  Srr,  in  order  to  adjust  the  probability  of  failure  in 
light  of  the  expected  technological  difficulty  of  the  project. 
The  existence  of  an  "optimistic  bias"  in  defense  contractor 
estimates  has  been  discussed  in  a  previous  section  of  this 
study.   This  final  adjustment  attempts  to  account  for  such 
bias  by  increasing  the  probability  of  failure  by  an  amount 
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which  is  dependent  upon  the  technological  advance  sought. 
This  would  appear  logical  since  if  no  technological  advance 
is  sought,  the  estimates  should  be  accurate  as  given,  while 
if  a  large  advance  is  sought,  the  probability  of  failure  can 
more  than  double. 

Once  the  Risk  index  has  been  generated,  it  becomes  de- 
sirable to  put  it  into  its  most  meaningful  and  useful  form. 
The  concept  of  Risk  can  be  thought  of  in  two  ways:   one  of 
which  is  negative  (;Z^^,  the  probability  of  program  failure), 
and  the  other  positive  (^rr,,  the  probability  of  program  suc- 
cess).  In  general,  a  statement  of  the  sort  "The  probability 
that  this  system  can  be  produced  for  X  dollars  in  Y  months  is 
0.63,"  would  be  preferred  to  the  corresponding  statement  of 
the  probability  of  failure.   Obviously,  the  two  measures  are 
complementary;  that  is,  }^^  =  1  -  ;2^,j,  =  1  -(1- ?^y)  (1+S^)   . 
If  the  value  ofjKj,   is  negative  (for  example,  if^^  or 
yr   =   0^  j^    =   -Srp),  it  should  be  considered  to  be  zero.   Since 
no  advanced  program  can  have  a  negative  value  for  S^,  ^  ^  is 
always  positive.   This  places  the  following  limits  on  the 
indices:   0  £  ^^  ^  1.0;   0  £:  /^  ^  1.0  .   The  Risk  index  is 
designed  to  support,  not  replace,  the  decision  maker.   Like 
any  tool,  it  should  be  used  with  common  sense  and  good  judg- 
ment.  A  small  value  for  >^  '^^^$  should  normally  eliminate 
a  project  from  consideration,  regardless  <!>f  its  y^     value. 
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Sensitivity  Analysis 

If  the  Success  index  C^™)  and  the  Risk  index  (  ^rp)  are 
expressed  in  terms  of  the  original  inputs,  it  can  be  seen 
that,  while  the  indices  may  be  imperfect,  they  do  respond 
in  a  quite  reasonable  manner  to  changes  in  the  input  para- 
meters, 

Jr^      =      1-(1-  f^    f^)   (1   +  S^)  ^^   =   (1-  j^^    ^$)  (1   +  S^)=   1-^^ 

0  ^  ]^^  ^  1.0  ;   0  :^  ^^  i^  1.0  ;   0  £  S^  :^  1.2 
Consider  the  change  in  ]^     for  the  following: 

(1)  An  increase  in  $  : 

An  increase  in  $  leads  to  an  increase  in  f^^ 

o  ? 

This  increase  in  1^ ^   leads  to  a  decrease  in  S 

+  ^;^^^  -A  (1  -  ^^^-J^^Xl  +  S^)  =  -A)Z5^  . 
A  decrease  in  S      implies  a  corresponding  increase 

in  ^T  • 

(2)  A  decrease  in  t  : 

The  decrease  in  t  leads  to  a  decrease  in  2^^ 

o  '  c 

which  in  turn  leads  to  an  increase  in  p   or  a 
corresponding  decrease  in  ^ 
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(3)     An  increase  in  S„: 

I 

If  a  change  in  configuration  or  in  one  of  the 
system  requirements  occurs  which  causes  S„  to 
increase,  this  will  cause  ^   to  increase  and)^ 
to  decrease  by  the  same  amount. 

+As^-^+A(i  -  ]^^r^)a  +  s^)  =  +^^^  =  -A]^ 

From  the  magnitudes  of  the  changes  in   „  brought  about 

by  changes  in  the  resource  constraints  ($  and  t  ),  the 

analyst  can  estimate  \^J^   and  '\  i   ^   .      These  estimates  may 

c)  $o     ^  to 
be  valid  over  greater  or  lesser  ranges,  depending  upon  the 

location  of  the  points  chosen  along  the  cumulative  distribu- 
tion function  of  Figure  8.   As  a  rule  of  thumb,  the  middle 
third  is  usually  fairly  linear  while  the  ends  are  definitely 
nonlinear.   If  he  can  then  determine  the  dollar  value  of  a 
schedule  decrease  in  the  program  (for  example,  if  he  is  told 
that  the  Air  Force  will  pay  an  additional  $2  million  for  a 
three-month  speed  up  in  the  time  table) ,  he  may  determine  the 
resultant  impact  upon  program  Risk  of  these  resource  constraint 
changes.  l£  it  is  desired  to  determined  the  impact  on  Risk  of 
a  three-month  reduction  in  program  length  and  a  $2  million 
budget  increase,  he  must  find 

/i^T=4rT|4t^  =  .3„„„,hs  ^^^T  I  ^^^  ,  +  ^2  million  " 

If  the  Risk  is  reduced  by  decreasing  time  and  increasing 
money  by  a  corresponding  amount,  this  indicates  that  the 
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schedule  may  profitably  be  reduced.   While  thlr.   aij.alysis  will 
not  necessarily  give  the  optimal  mix  of  time  and  funding  for 
a  project,  it  v/ill  3,ndicnte  the  proper  direction  for  improve- 
ment.  Efficient  strategies  can  be  determined  l.^y  varying 
the  values  of  $  and  t^  until  the  dollar  equivalent  of  a 
schedule  change  produces  an  equilibriujn  of  tho  form 

This  strategy  ir  efficient  because  the  amount  the  customer  is 
V7illing  to  increase  the  funding  for  a  decrease  in  schedule 
and  the  time  extension  he  is  willing  to  grant  for  a  decrease 
ill  cost  are  not  sufficient  to  reduce  the  Risk  associated  V73  th 
the  program. 

The  values  which  are  obtained  foi*  tlie  savi-itivities  of 
y^'r,   to  changes  in  resource  restrictions  \'ill  have  meaning  oo.ly 
over  limited  ranges.   If  significant  changes  are  contemplated, 
sensitivities  should  be  recoraputed  for  the  pioposed  values 
of  $^  and  ty. 

S  u mma  r y  an  d  C  on c  1  u  s  ijDn 
This  chapter  presents  a  discussion  of  the  reasons  behind 
the  selection  of  input  parameters  and  the  decision  to  use  a 
partially  subjective  evaluation  of  the  hisk  inde::.   The  steps 
followed  in  the  development  of  the  index  (sc.-:  Figure  9)  were; 
(.1)   Given  the  performance  requirements  of  a  system, 
generate  lov?,  high,  and  modal  values  for  t  and  $. 
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(2)  Using  thesa  estimates,  ccxistruct  the  probability 
(oi:  program  succesr)  disto-'ibutions  for  $  aiid  t. 

(3)  Cu  theae  distributions,  plot  $y  and  t^^  and  calcu- 
late the  areas'  to  the  left  of  these  constraints 

(  y^A  and  l^j.)  .   (Ultimately,  the  cumulative  distri- 
bution function  of  Figure  8  ma}'  be  generated  and 
the  probabilities  read  directly  from  the  ordinate.) 

(4)  Calculate  the  unadjusted  probability  of  success 

(5)  Adjust  the  probability  of  failure  for  state  of  the 
art  advance   ^^,  -  (1  ~  ;^]j)  (-  +  ^-p)  • 

(6)  To  sta-:e  the  Risk  dn  more  p'^citive  an<?^  usable  teirm.? ;> 
take  the  complement  of  p ^^  V7hich  is  the  probability 
of  total  program  success. 

yr^  - 1  "  (1  •^f\r^){i  -h  s^) 

(7)  To  apply  the  indeX;  generate  the  sensitivities. 


Ar-i 


At 


and  A]^v 


o 


-4$ 


o 


(8)   Determine  efficient  allocation  where 


A'j^T 


-1-  A  t. 


-  ^n 


TA 


o 


The  index  derived  here  is  not  an  ultimate  index  iii  any 
sense.   It  is  an  attempt  to  derive  e    lor:,ically  consistent 
methodology  for  generating  a  Risk  i-ade^r  vhich  has  some  real 
meaning  a-nrl  can  find  realistic  useful:. oto  in  decision  i.vnking 
Before  the  inde:-;  can  be  used  V7ith  any  degree  of  confidence. 
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it  must  be  validated  by  application  to  real  problems.   Al- 
though it  is  as  yet  an  untried  tool,  the  methodology  developed 
here  represents  a  step  forward  toward  a  measure  which  has 
physical  meaning  (probability  of  program  success  or  failure) . 
Additional  research  and  a  broader,  more  accurate  objective 
data  base  will  refine  or  revise  the  technique  suggested  here 
into  an  even  more  practicable  and  reliable  tool. 


.  CHAPTER  IV 
REFINEMENTS 

Revision  of  the  Present  Model 

In  cases  where  the  program  is  sufficiently  well  defined, 
alternate  methods  for  the  generation  of  the  cumulative  dis- 
tribution form  may  provide  a  more  solid  base  for  the  deriva- 
tion of  the  Risk  index.   One  such  method  involves  the  use  of 
concepts  developed  by  Faucett,  Henry,  and  Wilson  (19)  in 
their  Netv>7ork  Analysis  Model  (NAM)  .   Their  model  requires  that 
a  Program  Evaluation  and  Review  Technique  (PERT)  type  network- 
be  constructed  and  time  or  dollar  estimates  made  for  each 
activity.   A  computerized  model  then  makes  many  passes 
through  the  network  in  a  random  manner  and  generates  a  cumu- 
lative distribution  function  for  the  probability  of  completion 
versus  time  or  funding.   If  the  program  is  defined  in  suffi- 
cient detail  to  permit  this  type  analysis,  the  resulting 
distribution  may  give  a  more  accurate  picture  of  the  relation- 
ship which  actually  exists  between  cumulative  probability  of 
success  and  the  levels  of  funding  and  schedule.  (See  Figure  10.) 

A  second  method  which  may  find  application  in  reducing 
subjectivity  is  the  technique  known  as  the  Graphical  Evaluation 
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0 

and  Review  Technique  (GERT) (20) .   While  NAM  is  a  simulation, 
GERT  is  an  analytical  technique  in  which  topological  analysis 
permits  the  generation  of  the  moments  of  the  distribution 
which  describes  the  network  representing  the  program.   If  the 
form  of  the  distribution  can  be  recognized  from  these  moments, 
then  the  actual  distribution  can  be  generated  from  the 
moments.   Further  study  is  necessary  in  order  to  determine 
the  applicability  of  this  method  to  the  Risk  index  method- 
ology, but  the  approach  appears  to  have  promise. 

Another  technique,  which  may  be  used  on  projects  which 
are  not  defined  in  great  detail,  as  well  as  the  more  clearly 
detailed  cases,  is  the  one  presented  by  Sobel  (21).   He  shows 
that  while  the  PERT  technique  assumes  a  scaled  Beta  distribu- 
tion, the  ^specification  of  one  additional  piece  of  informa- 
tion (the  80  percent  central  range)  by  the  analyst  permits 
much  greater  freedom  in  the  form  of  the  frequency  distribu- 
tion and  in  general  results  in  distributions  which  more 
accurately  describe  the  actual  function.   Computer  programs 
are  developed  which  analyze  and  quantify  the  uncertainty  of 
cost  estimates.   While  Sobel 's  analysis  is  stated  only  in 
terms  of  cost,  it  could  be  applied  equally  well  to  time  dis- 
tributions.  Additional  effort  in  this  area  would  enhance  the 
validity  of  the  Risk  index  developed  here. 
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Two  recently  developed  decision  making  techniques  may 
lend  themselves  to  incorporation  into  the  sections  of  the 
Risk  procedure  which  require  subjective  analysis.   These  are 
DELPHI*,  developed  by  the  RAND  Corporation,  and  PATTERN^'^,  " 
developed  by  Honeywell's  Military  Products  Group.   The  two 
methods  are  very  similar  in  that  they  both  center  around 
reevaluation  of  decisions  upon  receipt  of  additional  infor- 
mation (the  opinions  of  other  experts  on  the  same  problem). 
The  basic  difference  is  that  DELPHI  forbids  discussion  between 
the  participants,  while  PATTERN  encourages  it.   Both  methods 
claim  to  have  demonstrated  that  in  most  cases  consensus  answers 
can  be  reached  after  several  iterations.   Techniques  such  as 
these  may  find  application  in  the  subjective  portions  of  the 
Risk  index,-procedure. 

Possible  Measures 

This  segment  of  the  study  lists  some  other  potential 
refinements  of  the  Risk  index  which  would  be  useful  in  the 


*Developed  under  United  States  Government  contract,  this 
technique  derives  its  name  from  "Project  DELPHI",  which'began 
in  the  early  1950 's  at  RAND.   The  initial  use  was  to  deter- 
mine vulnerability  of  the  United  States  to  nuclear  attack. 
It  has  since  been  refined  and  is  being  promoted  as  a  manage- 
ment planning  tool. 

Planning  Assistance  through  Technical  Evaluation  of 
Relevance  Numbers.   Developed  by  Honeyv/ell  to  aid  in  ranking 
proposals  in  terms  of  relative  attractiveness  for  future 
business . 
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assessment  of  program  proposals,  along  with  the  principal 
advantages  and  some  of  the  problems  associated  with  imple- 
menting each.   This  list  is  not  intended  to  be  complete,  but 
rather  to  indicate  the  wide  variety  of  possible  approaches  to 
the  problem  of  risk. 

Point  Estimate  of  the  Probability  of  Success 

The  measure  developed  in  this  study  may  be  classed  as 
a  point  estimate  of  the  probability  of  success.   Use  of  a 
logically  consistent  methodology  assures  some  degree  of  uni- 
formity among  values  assigned  by  different  analysts  and  per- 
mits extension  to  decision  making  and  assignment  of  tangible 
meaning  to  the  index.   The  principal  disadvantages  of  this 
sort  of  measure  lie  in  the  difficulty  of  generating  a  set  of 
data  in  the  proper  form,  the  fact  that  some  of  the  assumptions 
necessary  to  the  derivation  of  the  index  may  not  hold  strictly 
true,  and  the  inability  of  the  analyst  to  assign  a  confidence 
measure  to  his  estimate  of  the  probability  of  success. 

Point  Estimate  with  Standard  Deviation 

The  point  estimate  with  a  measure  of  dispersion  is  an 
approach  which  was  considered  at  some  length  during  this  investi- 
gation.  The  inclusion  of  a  dispersion  measure  would  permit  the 
analyst  to  indicate  a  confidence  band  about  the  value  he  obtains 
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for  the  index''^  and  would  allow  him  to  express  a  confidence 
level  for  the  data  from  which  the  estimates  were  derived. 
The  main  difficulty  arises  from  the  fact  that  the  form  of 
the  distribution  of  ]^  is  unknoxm  and  must  be  assumed.   The 
inability  to  derive  a  logically  sound  methodology  for  measur- 
ing the  standard  deviation  prevented  its  inclusion  in  the 
method  developed  here. 

Attainment  Profile 

The  attainment  profile  as  a  measure  of  risk  is  discussed 
in  a  following  section.   In  essence,  it  evaluates  the  proba- 
bilities of  attaining  various  percentages  of  the  desired 
schedule,  budget,  and  performance  goals.   An  advantage  of 
this  approach  is  that  by  use  of  a  "profile  adjustment  function' 
the  profile  may  be  updated  over  time.   This  permits  analysis 
of  a  more  dynamic  nature  than  is  possible  with  other  types  of 
risk  index.   Disadvantages  are  the  difficulty  encountered  in 
the  determination  of  the  form  of  the  initial  profile  and  the 
fact  that  the  entire  profile  must  be  updated  for  any  change 
in  inputs  or  specifications.* 


*This  could  take  the  form  of  a  probability  statement: 
P(.82  ^  Ft  ^  -96)  =  997o 

Subsequent  to  the  work  done  in  this  area  in  the  course 
of  this  study,  Terrell  (22,23)  independently  synthesized  an 
approach  similar  to  the  attainment  profile  for  a  study  of 
the  effects  of  schedule  compression  upon  program  risk. 
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Financial  or  Economic  Risk  Function 


The  financial  or  economic  risk  function  represents  a 
particularly  useful  tool  for  decision  makers,  since  the 
decision  to  attempt  a  project  ultimately  hinges  upon  economic 
considerations.*  This  method  would  generate  a  risk  function 
in  terms  of  the  impact  risk  has  upon  the  firm  from  an  economic 
point  of  view.   It  relieves  the  analyst  of  the  necessity  of 
making  a  "two-step"  study,  that  is,  first  assessing  the  risk 
and  then  applying  this  measure  to  financial  considerations. 
The  principal  disadvantage  stems  from  the  fact  that  it 
requires  simultaneous  consideration  of  several  aspects  of  a 
many-faceted  problem,  and  the  resulting  difficulty  necessi- 
tates the  use  of  even  more  sophisticated  analytical  techniques. 


Other  Possible  Measures 


There  are  many  other  possible  types  of  risk  indices 
which  may  be  developed,  depending  upon  the  particular  defi- 
nition of  risk  one  chooses.   One  such  general  type  of  index 
which  was  considered  in  the  course  of  this  investigation 
is  the  idea  that  risk  may  be  a  variance  (of  program  length 
or  funding  or  product  reliability,  for  example).   At  least 


*There  have  been  instances  in  which  military  values 
became  the  overriding  objectives,  but  even  under  these  con- 
ditions, the  decision  of  how  to  implement  the  desired  system 
becomes  one  of  economics. 
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one  other  study  (24)  speaks  of  risk  as  being  the  variance 
of  factors  which  are  not  included  in  the  estimating  equations. 
In  general,  the  term  "risk"  will  mean  different  things  to 
different  people  in  different  contexts,  and  the  type  of 
measure  which  best  describes  risk  is  a  function  of  the 
particular  definition  employed.   For  a  discussion  of  other 
measures  of  risk,  see  Some  New  Approaches  to  Risk  by  Byrne 
et  al.  (25). 

Problem  Areas 

Data  Accumulation 

One  major  problem,  regardless  of  the  approach  used  in 
the  risk  analysis,  is  the  unavailability  of  accurate  data 
in  usable  foirm.   The  form  of  government  data  can  obscure 
the  actual  contractor  costs;  there  is  little  usable  data 
exchanged  between  firms  because  of  the  competitive  nature 
of  the  industry;  firms  are  reluctant  to  provide  data  which 
indicate  that  they  have  overrun  cost  and/or  time  estimates; 
both  security  regulations  and  proprietary  motives  restrict 
this  flow  of  data. 

One  possible  solution  to  this  problem  is  the  generation 
of  a  data  base  by  conducting  a  series  of  controlled  experi- 
ments in  which  care  is  exercised  to  assure  that  accurate 
cost,  schedule,  and  performance  data  are  accumulated  in  a 
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usable  format.  This  could  be  carried  out  within  a  single 
firm  or  (if  precautions  were  taken  to  protect  the  parties 
involved)  among  firms.  Obviously,  all  firms  involved  might 
benefit  from  the  larger  data  base  available  as  a  result  of 
such  cooperation.  Certainly  the  availability  of  a  better 
file  of  historical  data  would  improve  the  accuracy  of  any 
risk  index. 

Bias 

The  problem  of  bias  is  one  which  appears  at  every  phase 
of  risk  analysis.   Several  studies  speak  of  the  "optimistic 
bias"  on  the  part  of  the  prospective  contractor  in  submitting 
estimates.   This  may  be  separated  into  deliberate  bias  and 
unintentional  bias.   The  deliberate  bias  is  injected  in 
order  to  underbid  competitors  and  obtain  a  contract,  knowing 
that  additions  will  permit  the  tentative  figure  to  be  raised 
to  a  more  realistic  one  after  the  program  has  begun.   Any 
point  in  the  analysis  which  permits  or  requires  judgment  on 
the  part  of  the  analyst  or  estimator  invites  bias  of  a  personal 
judgment  nature.   VJhile  seemingly  undesirable  at  first  glance, 
this  type  of  bias  may  well  be  a  desirable  feature  if  the 
analysis  makes  use  of  the  experience  of  senior  analysts. 

Another  type  of  bias  is  the  bias  in  a  statistical  sense 
which  is  present  in  the  selection  of  the  sample  of  data 
points  from  among  the  total  universe  of  all  aerospace  programs. 
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The  small  sample  size,  and  the  fact  of  nonrandomized  selec- 
tion of  points,  preclude  even  the  measurement  of  the  bias. 
This  sort  of  bias  has  been  unavoidable  in  the  past,  since 
the  data  which  were  available  in  usable  form  were  limited.  ' 
Hopefully,  this  situation  will  improve  over  time  as  the 
need  for  data  more  applicable  to  risk  analysis  becomes 
recognized  throughout  the  industry. 

Distribution  Forms 

At  various  places  in  the  analysis  of  Risk,  it  becomes 
necessary  to  assume  forms  for  statistical  distributions. 
In  most  instances  the  lack  of  data  makes  justification  of 
the  assumption  difficult.   A  case  in  point  is  the  assumption 
of  the  beta  distribution  to  describe  the  distribution  of 
the  probabilities  of  completing  a  program  within  certain 
resource  constraints.   The  primary  justification  given  was  that 
a  previous  study  (PERT)  had  assumed  this  distribution  form 
for  a  similar  parameter  with  good  results.   One  reason  for 
the  inability  to  generate  a  measure  of  dispersion  to  accompany 
the  Risk  index  of  this  study  was  the  inability  to  justify 
the  choice  of  distribution  forms.   The  only  solution  to  this 
problem  is  the  accumulation  of  sufficient  data  to  enable 
analysts  to  determine  the  actual  forms.   Lacking  this  data 
bank,  one  may  oniiy  rely  upon  logic  and  pragmatic  results  for 
justification  of  assumed  distributions. 
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Nonuniform  SOA-A  Measures 

At  present  there  appears  to  be  no  methodology  available 
for  measuring  SOA  advances  that  does  not  involve  a  substan- 
tial amount  of  judgment  or  subjective  measurement.   While 
the  use  of  subjective  judgment  has  the  advantage  of  permitting 
the  experienced  analyst  to  utilize  the  knowledge  acquired 
over  time,  it  does  preclude  the  possibility  of  specifying  a 
single  nxomber  which  would  represent  the  SOA-A  for  a  program. 
At  the  present  time  there  is  an  array  of  SOA-A  measures  rather 
than  a  single  number;  there  is,  in  general,  no  consensus  among 
analysts  as  to  the  "true"  value.   There  are  advantages  and 
disadvantages  to  this  arrangement,  but  the  problem  does 
exist.   Chapter  II  contains  a  more  thorough  discussion  of 
this  problem. 

Cost-Time  Tradeoffs 

In  the  analysis  presented  in  this  study,  as  in  prior 
studies,  it  was  necessary  to  make  the  simplifying  assumption 
that  cost  and  development  time  are  independent  parameters. 
In  fact,  the  economist  is  aware  that  a  tradeoff  relationship 
exists  between  these  inputs  similar  to  that  depicted  in 
Figure  11.   In  general,  the  function  will  be  different  for 
each  program  and  will  change  over  time  during  the  program. 
The  tradeoffs  may  be  expressed  at  a  point  or  over  a  small 
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segment  of  the  curve  at  a  given  time,  but  these  are  sensitive 
to  political,  social,  and  military  attitudes  and  thus  may 
change  rapidly  and  radically.   As  more  sophisticated  tech- 
niques are  brought  to  bear  upon  the  problem,  the  cost-time  " 
tradeoff  function  can  be  defined  and  periodically  updated 
to  permit  the  development  of  a  more  refined  risk  index. 


E 


Cost  ($) 


Figure  11 

TIME -COST  TRADEOFF  CURVE 
(Reference  12,  page  510) 
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Summary 


This  chapter  has  been  devoted  to  a  discussion  of  some 
possible  refinements  in  the  Risk  index  developed  in  Chapter 
III.   Several  techniques  which  can  be  of  value  in  reducing 
subjectivity  in  certain  areas  of  the  study  (NAM,  GERT,  DELPHI, 
PATTERN,  and  Sobel's  approach)  are  presented.   The  difficulty 
associated  with  the  use  of  NAM  or  GERT  arises  from  the  ne- 
cessity of  having  the  program  defined  to  a  level  of  detail 
which  is  unrealistic  for  most  advanced  programs.   The  main 
barrier  to  use  of  the  iterative  judgment  techniques  is  that 
these  require  carefully  structured  batteries  of  questions, 
the  cooperation  of  several  experienced  analysts,  and  greater 
amounts  of  time  than  may  usually  be  available.   Further  analy- 
sis  may  reveal  that  a  modification  of  one  of  these  general 
approaches  will  prove  fruitful. 

In  analysis  of  the  Risk  index  of  Chapter  III,  several 
types  of  risk  measurement  indices  which  would  be  of  value  in 
evaluating  aerospace  program  proposals  are  treated.  Some  of 
these  are  essentially  refinements  of  the  m.odel  developed  in 
this  study,  while  others  represent  different  ways  of  looking 
at  risk  and  the  application  of  different  sorts  of  analytical 
tools.   A  final  section  outlines  some  of  the  problems  which 
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face  the  analyst  in  attempting  to  measure  the  risk  associated* 
\;ith  a  particular  aerospace  program. 


CHAPTER  V 
EXTEKSIOi.^  TO  DECISION  MAKING 

Purpose 

The  purpose  of  this  chapter  is  to  indicate  a  means  for 
the  application  of  the  Risk  index  to  the  problem  solving  or 
decision  making  process.   First,  the  various  environments 
in  v^hich  decisions  are  made  are  presented;  then  there  follov7S 
a  brief  discussion  of  the  elements  of  game  theory  with  ex- 
tensions to  utility  theory  and  the  application  of  the  Risk 
index  in  both  of  these  areas;  finally,  a  section  is  devoted 
to  explanation  of  various  types  of  contracts  and  the  re- 
lationships which  may  exist  betv7een  Risk  and  contract  type. 

Management  peci s ions 

Decision  Making  • 

Economists  usually  distinguish  three  broad  categories 
or  classes  of  decisions:  decisions  under  conditions  of 
certainty,  decisions  under  conditions  of  uncertainty^ ,  and 


"For  a  good  discussion  of  the  difference  between  uncer- 
tainty and  risk,  see  Knight  (2) . 
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decisions  under  risk»"  The  classifications  are  dependent 
upon  the  confidence  the  decision  maker  may  place  in  the  data 
which  he  used  as  a  basis  for  the  decision. 

One  of  the  clearest  V7a5''s  to  differentiate  among  these 
three  situations  is  to  consider  a  hypothetical  aerospace 
contractor.   He  has  the  capability  of  buildiiig  either  long 
range  military  aircraft  (bombers)  or  intercontinental  ballis- 
tic missiles  (ICBM's),  but  not  both,  and  finds  that  he  must 
decide  which  of  these  tV70  product  lines  he  will  produce.   Due 
to  the  long  lead  times  required,  he  cannot  wait  until  the 
defense  department  issues  a  request  for  proposals  on  the  nev; 
systems  to  be  ordered  but  must  decide  now  upon  his  course  of 
action  for  several  years  to  come. 

If  the  company's  president  has  a  son-in-law  vAio   is 
Under  Secretary  of  Defense  and  has  access  to  the  information 
that  the  Secretary  of  Defense  has  decided  to  rely  exclusively 
upon  procurement  of  ICBM's  for  the  nation's  defense  in  the 
coming  years,  the  company  can  decide  with  certainty  (or  as 

"The  term  "risk"  as  used  in  this  section  does  not  refer 
specifically  to  the  same  concept  as  the  term  "Risk"  used  in 
previous  sections.  Although  the  connotations  are  similar,  the 
term  "risk"  will  be  used  to  refer  to  the  general  concept  and 
the  capitalized  "Risk"  to  the  more  narrowly  defined  concept 
of  this  report o 
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close  to  certainty  as  is  possible  in  the  defense  industry) . 
If  the  president  were  not  so  fortunate  in  his  daughter's  choice 
of  suitors,  the  firm  v7ould  find  itself  operating  vmder  condi- 
tions of  uncertainty  -  having  no  idea  V7hich  course  of  action 
it  she  ild  pursue.   In  the  absence  of  any  information  about 
the  relative  likelihoods  of  the  possible  defense  postures  to 
be  ass.med  by  the  defense  department,  the  game   theorists 
suggest  that  one  possible  wa}''  to  make  the  decision  is  to 
assume  equal  probabilities  of  occurrence  for  each  and  fl5.p  a 
coin. 

There  is,  hoxi/ever,  a  middle  ground  between  secure  knov7- 
ledge  and  complete  bev7ilderment  on  the  part  of  the  decision 
maker.   This  area  will  be  referred  to  as  decision  making  under 
conditions  of  risk.   In  the  example,  the  firm  could  move  from 
conditions  of  uncertainty  to  those  of  risk  by  searching  the 
available  literature,  hiring  people  from  the  defense  depart- 
ment, performing  analyses  of  the  conditions  which  would 
determine  the  future  defense  posture  (eog.,  analysis  of  the 
enemy  threat  in  the  future) ,  and.  assigning  probabilities  of 
occurrence  to  the  possible  alternatives.   Having  done  this, 
the  decision  maker  is  said  to  be  deciding  under  conditions 
of  risk. 
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To  summarize,  if  the  decision  maker  knows  the  consequen- 
ces of  the  o.lternative  courses  of  action  open  to  him  at  the 
time  of  the  decision,  he  decides  under  certainty;  if  he 
knov7S  the  possible  outcomes  and  can  assign  a  set  of  proba- 
bilities  to  them,  he  decides  under  risk;  if  he  ca.nnot  assigi"> 
probabilities  to  the  possible  outcomes  or  if  he  does  not  know 
what  outcomes  may  occurj  he  decides  under  conditions  of  un- 
certainty.  By  assigning  a  probability  of  success  to  proposed 
programs,  this  report  attempts  to  enable  the  decision  maker 
in  the  aerospace  industry  to  move  from  decision  making  under 
conditions  of  uncertainty  to  decision  making  under  risk. 

Gasie  Theory^':"  and  IJtili.ty 

One  area  in  V7hich  the  Risk  index  may  find  direct  appli- 
cation as  a  decision  makiiig  tool  is  game  theory.   This  is 
a  widely  known  tool  for  quantitative  decision  inaking  which, 
in  recent  years,  has  come  into  wide-spread  use  as  a  method 
for  m.aking  business  decisions.   Since  this  report  attempts 
to  enable  the  decision  maker  to  move  from  the  area  of  un- 
certainty to  that  of  risk,  it  is  perhaps  instructive  to 
examine  this  quantitative  decision  m'.king  tool. 


"For  a  thorough  analysis  of  this  subject,  the  reader 
is  referred  to  Chernoff  and  Moses  (26),  Luce  and  Raiffa  (27), 
Swalm  (28),  Voii  Neumann  and  Morgans  tern  (29),  end  Williams 
(30). 
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111  moving  into  the  area  of  decision  making  under  condi- 
tioas  of  risk,  the  analyst  finds  it  necessary  to  assign  pro- 
babilities  of  occurrence  to  the  various  possible  outcoraes  of 
a  proposed  program.   One  method  of  accomplishing  this  is  by 
direct:  substitution  of  the  Risk  index  of  this  study  ((^'p)  for 
the  piobability  that  the  project  will  be  successful,  A  more 
sophisiiicated  analysis  V70uld  also  consider  the  value  of  com- 
pleting 50  or  75  percent  of  the  project  (usually  this  V7ould 
result  in  a  negative  payoff  or  loss)  and  techniques  similar 
to  the  attainment  profile  could  be  applied  to  generate  the 
probabilities  of  occurrence  of  the  respective  events.   The 
same  sort  of  analysis  can  be  applied  to  a  utility  function, 
once  that  function  has  been  generated. 

Attainment  P r o file'''-' 

The  discussion  to  this  point  has  considered  Risk  in  terms 
of  the  probability  of  completing  100  percexit  of  the  program 
goals,  A  more  thorough  analysis  would  consider  the  entire 
life  of  a  project  and  the  probability  of  completing  various 
percentages  of  the  total  program  requirements  within  the 


*Much  of  the  initial  work  on  the  attainment  profile  con- 
cept was  done  by  VJ.  J.  Bailey  III  of  General  Dynamics'  Fort 
Worth  Division  ivi  an  unpublished  paper.  The  author  gratefully 
atcknowledges  Mr.  Bailey's  assistance  in  the  preparation  of 
this  report. 
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given  resource  restrictions.  A  technique  which  attempts  this 
sort  of  analysis  is  the  attainment  profile. 

Programs  with  equal  probabilities  of  completing  100  per- 
cent of  the  performance  goals  can  have  significantly  different 
profiles;  conversely,  projects  with  similar  profiles  may  differ 
with  respect  to  their  probabilities  of  100  percent  completion. 
Consider  the  profiles  of  Figure  12.  Profiles  I  and  II  each 
have  y/^  values  of  10  percent;  yet  the  probability  of  completing 
90  percent  of  project  I  is  only  15  percent.  Project  I  could 
therefore  entail  much  greater  risk  than  project  II.  Project 
III  has  a  m.uch  larger  j!/^  than  project  II,  but  it  may  actually 
involve  more  risk. 

It  is  possible  to  derive  a  profit'-^  function  which  states 
profit  earned  by  the  contractor  as  a  function  of  the  percent 
of  project  goals  completed.   If  such  a  function  were  defined 
and  superimposed  on  the  attainment  profile,  the  expected  value 
of  profit  and  the  probability  of  achieving  any  given  level  of 
profit  (or  loss)  can  be  generated.   Such  values  could  be  used 
as  inputs  to  financial  planning  models  or  game  type  decision 
models,  and  should  generally  tend  to  result  in  a  more  thorough 
analysis  than  the  simpler  Risk  index  developed  in  Chapter  III. 


*The  term  "profit"  as  used  in  this  section  refers  to 
accounting  profit  (income  less  cost)  rather  than  to  economic 
profit. 
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It  is  also  theoretically  possible  to  derive  a  utility 
function  for  the  firm  and  superimpose  it  on  the  attainment 
profile.  While  this  is  a  theoretical  possibility,  the  prac- 
tical realization  of  such  a  curve  is  doubtful.  Profit  functions 
relative  to  percent  of  completion  are  very  difficult  to  define, 
and  utility  is  not  linear  with  profit  and  can  change  over  time 
with  shifts  in  government  policy,  tax  legislation,  corporate 
size  and  image  considerations,  and  other  highly  subjective 
factors o   In  short,  the  development  of  a  utility  function  for 
use  with  the  attainment  profile  is  not  practical  at  the  present 
time , 

Assuming  a  linear  profit  function,*  Figure  13  indicates 
some  of  the  uses  of  the  attainment  profile  concept  as  an  ana- 
lytical tool.  J^^  is  the  probability  of  completing  100  percent 
of  the  goals,  and  P  is  the  dollar  value  of  the  profit  which  is 
expected  if  all  requirements  are  met  within  the  limits  set. 
Point  B  is  the  breakeven  point;  if  this  fraction  of  the  re- 
quirements are  met,  the  firm  will  neither  gain  nor  lose  money. 
P(B)  is  the  probability  of  attaining  this  level  of  completion. 
Point  M  marks  the  70  percent  completion  level.   Point  L  shows 
the  expected  amount  of  loss  for  this  amount  of  completion,  and 
P(M)  is  the  probability  that  this  level  can  be  realized. 


*Over  the  relevant  ranges  of  completion,  this  assumption 
is  not  unreasonable.   Incentive  contracts  purposely  define  a 
linear  return  in  the  area  of  the  contractual  figures 
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The  construction  of  an  attainment  profile  is  not  a 
well-defined  process  at  presento   One  possible  approach  is 
to  rely  upon  the  expertise  of  the  analyst;  another  is  to 
use  an  approach  similar  to  the  Risk  index  method  to  quantify 
the  1  robability  of  success  at  several  levels  of  co.npletion 
and  connect  the  points  thus  generated  to  form  the  profile; 
a  third  approach  would  utilize  historical  data  on  similar 
projects;  a  fourth  method  x^ould  be  some  combination  of  the 
first  three  approaches. 

The  flow  diagran  of  Figure  14  displays  a  procedure  for 
developing  and  applying  the  attainment  profile  concept  in 
aerospace  planning.   The  program  parameters  are  input  into 
a  NAM  or  GERT  model.   The  outputs  from  the  model  are  used 
to  gen-jrate  the  initial  attainment  distribution.   This 
initial  profile  is  then  adjusted  by  the  evaluation  of 
subjective  factors  and  historical  data.   The  entire  profile 
can  then  be  periodically  updated  or  revised  as  conditions 
change  over  time. 

In  summary,  this  section  has  discussed  the  concept  of 
the  attainment  profile;  its  possibilities,  shortcomings, 
and  some  of  the  difficulties  associated  with  its  development 
are  suggested.  As  a  concept  which  exhibits  great  potential 
as  a  decision  making  tool,  its  value  is  obvious;  as  a 
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practically  attainable  maasure  for  aerospace  programs,  its 
future  is  less  certain.  As  further  research  gives  greater 
insight  into  the  nature  of  risk  and   its  consequences  to 
program  success,  the  attainment  profile  may  becorae  a  pov7arful 
manage  aent  tool. 


Contracting  and  Risk"^' 


Tie  area  of  contract  provisions  vras  originally  investi- 
gated X  ith  the  intent  of  determining  how  these  provisions 
reflected  or  m.easured  contractor  efficiency.   It  was  hoped 
that  suoh  an  investigation  would  generate  factors  that  could 
be  used  to  debias  or  normalize  cost  data  for  individual 
firms*  efficiency  or  lack  of  it.   This  original  line  of 
investigation  V7as  not  productive.  The  analysis  of  contract 
provisions,  hov7ever,  did  shed  light  on  the  relationship 
bettv^een  contract  provisions  and  Risk.   Even  more  important, 
the  investigation  suggested  the  possible  application  of  a 
Risk  index  as  an  aid  in  establishing  contract  types  and 
contract  provisions.   Currently  the  techniques  used  in  con- 
tract procurement  are  still  being  developed  and  contract 
terms  are  determined  largely  on  the  basis  of  rules  of  thumb 


*The  author  gratefully  acknowledges  the  assistance  given 
by  Mro  Morris  L.  Williamson,  Jr.  of  the  Fort  Worth  Division 
of  General  D^'nam/ics  in  the  preparation  of  this  section. 
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and  judgment .  AFS_CM  375_Sy s tera  yZr^glp.e.eri.Vi^   Managp.men L 
Procedures  (31)  represents  an  attoiupt  by  the  Departnient  of 
Defense  to  systeuiize  the  systems  developracnt  avx\   contracting 
procedure.   As  this  system  and  others  sprailar  in  nature  are 
developed,  the  contracting  procedure  v/ill  become  more  well 
defined. 

Selection  of  conti^act  provisions  is  essentia.lly  c   pro- 
cess by  which  financial  or  cost  risk  is  allocated  betv7een 
the  contracting  aL,e'ncy  and  the  contractor.   it  should  be 
kept  in  miiid  that  a  project's  schedule  risk  and  performance 
risk  cannot  be  allocated;  the  contracting  agency  always 
bears  ultinate  responsibility  for  these  risks.   The  con- 
tracting agency,  however,  can  minimize  these  risks  by  estab- 
lishing contract  provisions  v/hich  stimulate  the  contractor 
to  meet  schedule  and  perfcnnance  requirements. 

The  follov.'ing  basic  classifications  of  contract  types 
were  obtained  from  Moore  (32) : 

I.   Cost 

a.  CPFF  =-■  Cost  plus  fixed  fee  ■ 

b.  FPR-E  -   Fixed  price  rede terminable 
(retroactive) 

II.      Incentive 

a.  FPI   -  Fixed  price   iv»ceni:ive   (including 
pcrform^ance  incentives) 

b.  CPIF  --  Cost   plus   inccuitive   fee 

c.  FPri-A  ■■'  Fixed   price  redeterminable 
(prcospective) 

d.  FF?  --  Fir;.'   fixed    price  with   (\vithout) 
escalation 
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More  explicityly,  the  following  conditions  characterize  cost 
and  incentive  type  contracts: 

CPFF:   Established  on  an  estimate  of  total  cost 

that  the  contractor  will  incur  plus  a  fixed 
fee  (usually  a  percentage  of  total  cost) . 
Provisions  state  that  the  fee  does  not 
change  although  costs  may  increase. 

FPR-E:  Provides  for  a  fixed  price  for  certain 

periods  of  time,  but  provides  for  retro- 
active price  redetermination. 


FPI 


CPIF: 


A  negotiated  target  cost  and  target  profit 
is  determined.  A  price  ceiling  and  a  for- 
mula for  adjusting  the  final  price  and 
profit  are  also  defined.   (Ceilings  and 
formulae  provide  the  incentive  for  con- 
tractors to  reduce  costs  and  to  share  in 
the  savings.) 

An  estimated  target  cost  and  a  target  fee 
is  determined.   A  formula  for  fee  adjust- 
ment (proportional  to  under/over  run)  is 
set.   Similar  to  FPI  except  there  is  no 
separate  price  ceiling  on  a  CPIF  contract. 


FPR-A:  Price  is  fixed  initially  for  some  interval 

of  time  and  then  reset  at  intervals  for  each 
future  period.   Similar  to  a  series  of  FFP 
contracts  for  each  interval, 

FFP:   Provides  for  delivery  of  a  product  or  ser- 
vice for  a  price  specified  in  the  contract. 

CPAF:   (Cost  plus  award  fee)   The  contractor  is 

awarded  a  fee  which  is  based  upon  his  per- 
formance in  managing  the  program  and  meet- 
ing the  requirements  as  they  evolve.   This 
type  contract  is  not  mentioned  by  Moore, 
but  has  been  used  recently  in  contracts 
which  involve  advanced  technology  and  re- 
quire frequent  adjustment  of  specifications. 

Within  the  fraraev7ork  of  the  various  contract  types, 

efficiency  in  procurement  and  contracting  is  partially 
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determined  by  risks  and  incentives.   More  specifically,  the 
incentive  rates  are  determined  to  some  degree  by  the  inherent 
technological  uncertainty  involved  in  the  particular  program. 
When  the  initial  assessment  of  technological  uncertainty  is 
high,  management  often  attempts  to  prolong  contract  negotia- 
tions.  This  prevents  prematurely  fixed  target  prices,  which 
are  often  unrealistic  and  lead  to  cost  overruns. 

In  most  advanced  programs,  technical  uncertainty  de- 
creases with  program  progress.   The  contractor  being  forced 
into  relatively  early  commitments  must  anticipate  future 
areas  of  difficulty  and  incorporate  these  uncertainties  into 
the  contract  essentials  (i.e.,  varying  the  profit  rates, 
clauses  for  future  renegotiations) .   The  inclusion  of  terms 
such  as  these  decreases  the  inherent  financial  risk  to  the 
contractor.   Therefore,  one  of  the  m.ain  purposes  of  the 
contract  is  to  serve  as  an  allocater  of  financial  risk  be- 
tween the  contractor  and  the  procurer. 

The  firm  fixed  price  contract  places  maximum  financial 
and  technological  risks  upon  the  contractor  and  therefore 
induces  a  maximum  incentive  for  cost  control.   The  fixed 
price  contract  provides  a  minimum  cost  Risk  for  the  customer 
since  the  maximum  amount  to  be  paid  is  fixed  during  negotia- 
tions.  This  contract  type  is  used  for  low  SOA-A  type  pro- 
grams or  when  reasonably  definite  design  or  performance 
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specifications  are  available.   When  the  cost  estimating  uncer- 
tainties are  not  minimal  or  when  uncertainties  surrounding 
the  contract  performance  cannot  be  evaluated,  consideration 
is  given  to  other  contract  types.   For  example,  by  the  addi- 
tion of  escalation  clauses,  the  contractor's  financial  risk 
in  the  fixed  price  contract  is  reduced.   As  one  progresses  to 
the  fixed  price  with  incentive  clauses,  the  total  program 
risk  is  distributed  betvjeen  the  procurer  and  contractor  who 
share  the  responsibility  for  costs  greater  or  less  than  the 
original  estimate. 

The  cost  plus  incentive  fee  contract  is  negotiated  when 
it  is  highly  probable  that  the  development  is  feasible  and 
the  desired  performance  objectives  have  been  determined. 
This  type  of  contract  does  not  change  the  technical  uncer- 
tainty connected  with  a  program,  but  it  may  indirectly  in- 
crease the  probability  of  succeeding.   This  is  accomplished 
by  providing  limited  incentives  (and  penalties)  and  thus 
stimulating  the  contractor  to  be  more  efficient  in  his 
efforts  and  to  manage  the  contract  effectively.   This  type 
of  contract  may  be  best  suited  to  programs  of  moderate 
complexity  and  technological  advance. 

The  cost  plus  a  fixed  fee  contract  is  a  cost  reimburse- 
ment contract  with  a  fixed  fee  for  the  contractor  independent 
of  the  management's  ability  to  control  costs.   This  implies 
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the  financial  risk  to  the  contractor  is  almost  zero.   Hence, 
this  type  of  contract  has  been  used  in  high  complexity  pro- 
grams where  the  SOA-A  is  great,  the  level  of  effort  required 
is  unknovm,  and  the  relative  probability  of  success  is  un-  " 
determined. 


Figure  15  illustrates  the  general  relationship  betwe 
the  particular  contract  type  desired  by  the  contractor  and 
the  program's  technological  risk. 
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Figure  15 
CONTRACT  TYPE  VERSUS  PROGRAM  RISK 
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Although  contract  form  alone  does  not  explain  the  final 
performance  of  a  program,  it  is  an  important  factor  in  de- 
termining the  size  of  cost  and  schedule  over/under  runs. 
Although  the  population  of  DOD  contract  procurement 
actions  per  year  numbers  in  the  thousands,  most  of  the 
samples  investigated  by  recent  researchers  have  been  rela- 
tively small  and  stratified  in  type.   Because  of  this  non- 
random  selection,  inferences  drawn  cannot  be  justified  in 
the  statistical  sense.   However,  some  general  trends  from 
past  programs  can  be  obtained  from  the  studies  by  Moore, 
Marshall  and  Meckling,  and  Peck  and  Scherer. 

The  study  by  Moore  concludes  that  on  the  cost  type 
contracts,  the  average  profit  rate  ranges  from  6.3  percent 
to  6.8  pergent.   On  the  other  hand  profit  rates  of  incentive 
contracts  are  higher  by  a  sizeable  amount,  averaging  around 
8  percent  profit.   The  rationale  for  the  difference  is  that 
there  is  a  higher  financial  risk  attached  to  the  incentive 
contracts  and  that  a  higher  reward  must  be  offered  to  induce 
the  contractor  to  be  efficient. 

For  example,  from  the  sample  of  228  incentive  contracts 
investigated  by  Moore,  one  finds  that  on  the  average  the 
apparent  efficiency  probabilities  for  an  under/over  run  are: 

« 

0.74  for  a  cost  underrun 

0.50  that  the  cost  underrun  is  0  -  10  percent 
0.69  that  results  will  be  within  +  10  percent  of  the 
target  costs 
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Care  must  be  taken  in  the  interpretation  of  the  above 
results.   Although  there  is  indicated  a  0.74  probability  of 
a  cost  underrun,  one  should  not  conclude  that  the  procure- 
ment of  an  incentive  contract  for  a  program  will  indicate  " 
that  program's  success  with  the  above  stated  probability. 
It  must  be  kept  in  mind  that  the  type  contract  negotiated 
is  usually  indicative  of  program  complexity  and  uncertainty, 
and  that  the  process  of  contracting  is  a  complex  procedure. 
As  pointed  out  by  Moore,  "The  basic  rationale  for  renegotia- 
tion is  the  existence  of  uncertainty  in  defining  what  is 
being  contracted  for  and  uncertainty  in  defining  and  measuring 
performance. "* 

In  another  perspective,  Marshall  and  Meckling  in  their 
research  came  to  the  follov/ing  general  conclusions: 

(1)  "Early  estimates  of  important  parameters  are 
usually  quite  inaccurate.   They  are  inaccurate 
in  two  respects.   First,  such  estimates  are 
strongly  biased  toward  overoptimism.   Second, 
aside  from  the  bias,  the  errors  in  estimates 
evidence  a  substantial  variation.   That  is, 
even  if  estimates  were  multiplied  by  an 
appropriate  standard  factor  to  eliminate  the 
bias,  a  non-negligible  source  of  error  re- 
mains . 

(2)  The  accuracy  of  estimates  is  a  function  of  the 
stage  of  development,  i.e.,  estimates  improve 
as  development  of  the  item  progresses.   This 
also  means  that  estimates  for  development  pro- 
jects representing  only  modest  advances  tend 

to  be  better  than  for  more  ambitious  projects."*^ 


*Ref.  27,  p.  120 


'Ref.  11,  p.  1. 


-  89  - 

In  the  earliest  stages  of  research  and  exploratory 
development,  there  are  little  significant  data  available. 
Here,  use  of  a  CPFF  or  general  cost  contract  may  be  neces- 
sary because  of  inability  to  analyze  the  magnitude  of  tech- 
nical or  performance  uncertainty.   However,  once  more 
definitive  data  for  a  particular  program  are  available, 
other  contract  types  with  associated  incentive  provisions 
are  possible.   The  Risk  index  facilitites  establishment  of 
these  provisions.   Risk  measurements  of  the  prospective 
program  plus  Risk  measurements  and  outcomes  for  previous 
programs  will  aid  in  determining  whether  cost  plus  incentive 
fee,  fixed  price  incentive,  or  some  other  form  is  most 
appropriate.   In  addition,  with  the  use  of  an  index  and  a 
game  theory  formulation,  the  establishment  of  incentive 
provisions  will  be  placed  on  a  firmer  foundation.   With  the 
index,  the  probabilities  of  meeting  cost,  schedule,  and 
performance  objectives  contained  in  the  provisions  may  be 
assessed  in  a  consistent  quantitative  manner. 


CHAPTER  VI  • 

RISK  INDEX  APPLICATION 

Introduction 

One  way  to  promote  a  more  thorough  understanding  of 
the  Risk  index  methodology  is  through  the  use  of  an  example, 
Due  to  the  classified  nature  of  much  of  the  cost  and  per- 
formance data  on  government  aerospace  programs,  it  is  not 
possible  to  present  an  actual  program  as  an  example  of  the 
use  of  the  Risk  index.   It  was,  therefore,  deemed  necessary 
to  generate  a  hypothetical  example  program  in  order  to 
demonstrate  the  use  of  the  method.   It  should  be  pointed 
out  that  for  the  purposes  of  this  analysis  the  question  of 
whether  or  not  the  firm  receives  the  contract  will  not  be 
considered.   The  aim  is  rather  to  evaluate  the  feasibility 
of  bidding  on  the  program  under  various  combinations  of 
schedule  and  budget. 

Specifications 

The  design  and  performance  specifications  shown  in 
Table  3  for  a  jet  fighter  aircraft  will  be  taken  as  the 
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definition  of  the  program.   In  addition  to  these  specifica- 
tions, the  avionics  systems  must  include  terrain -following 
radar;  the  capability  for  multiple  launch  and  tracking  of  the 
latest,  most  advanced  missiles;  infrared  optical  system;  arid 
provisions  for  all-weather,  day-night  operation.   The  Depart- 
ment of  Defense  indicates  that  it  expects  that  if  the  develop- 
ment program  is  successful,  there  will  probably  be  an  order 
of  approximately  1000  units. 

Table  3 
DESIGN  AND  PERFORMANCE  SPECIFICATIONS 


Maximum  Speed  at  Optimal  Altitude 

Maximum  Takeoff  Weight 

Gross  Weight  (Empty) 

Maximum  Thrust  at  Sea  Level  (total) 

Combat  Radius  (Supersonic) 

Ferry  Range 

Number  of  Engines 

Wing  Span 

Length 


Y^^   and  ]//g  Development 


Mach  3,  1720  knots 
60,000# 
30,000# 
50,000# 
1000  miles 
4000  miles 
2 
50' 
60' 


The  advanced  projects  cost  analysis  group  of  the  company 
develops  estimates  for  the  costs  of  the  program.   These  are 
presented  in  Table  4. 
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Table  A 


INITIAL  COST  ESTIMATES  BY  COST  CATEGORY 


Cost  Item 

Total 
RDT&E  Cost 
$Million 

Average 

Production  Cost 

Per  Aircraft 

0  1000  Units 

$Million 

Total  Program 
Cost 
$Billion 

Airframe 

Engines 

Avionics 

Ground  Support 
Equipment 

Initial  Spares 

400 
600 
200 

20 

3.0 
1.0 
2.0 

.5 

1.5 

3.40 
1.60 
2.20 

.52 

1.50 

TOTAL 

1,220 

8.0 

9.22 

These  numbers  will  be  taken  as  the  most  likely  (modal) 
estimates.   The  cost  analysts  researched  the  available  his- 
torical  data  and  determined  the  low  and  high  1  percent  pro- 
bability estimates.   These  optimistic  and  pessimistic  estimates 
are  shown  in  Table  5. 

This  program  can  be  considered  in  many  different  ways. 
A  Risk  index  for  each  of  the  major  cost  categories  could  be 
developed;  an  index  for  the  total  program  cost  could  be 
generated;  the  RDT&E  program  can  be  considered  separately  from 
the  production  program;  both  can  be  considered  as  a  unit  for 
each  category;  or  the  sum  of  all  the  categories  over  the 
total  program  can  be  defined  as  the  program  under  consideration. 
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Here  the  program  will  be  considered  in  two  ways:   first,  the 
RDT&E  program  separately;  second,  if  there  appears  to  be  a 
reasonable  chance  of  successfully  developing  an  acceptable 
aircraft,  the  whole  program  (both  RDT&E  and  Production)  will 
be  considered  as  a  package. 

The  data  from  Table  5  and  the  PERT  assumptions  (X  = 

g )   were  used  to  generate  the  cumulative  distributi 

functions  of  Figures  16  and  17  for  the  RDT&E  and  total  pro- 
grams, respectively.   Similarly,  the  time  estimates  of 
Table  6  were  used  to  determine  the  cumulative  time  distri- 
butions of  Figures  18  and  19. 


on 


Table  6 

OPTIMISTIC,  MODAL,  PESSIMISTIC,  AND 
ESTIMATED  MEAN  PROGRAM  SCHEDULE  ESTIMATES 


Program 

Optimistic 
Estimates 
a 

Modal 

Estimates 

m 

Pessimistic 

Estimates 

b 

Estimated 

Mean 

/v  -  a+4m-hb. 
(X   -.  g   ) 

RDT&K  Only 

27  months 

48  months 

69  months 

48  months 

RDT&E  and 
Production 

5.5  years 

9  years 

14  years 

9.25  years 

The  method  used  to  generate  the  cumulative  distribution 
functions  was  admittedly  a  crude  one.   The  only  known  points 
on  the  curve  are  the  mean  value  and  the  1  percent  tails. 
Other  points  must  be  determined  from  graphical  interpretation 
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of  areas  under  the  appropriate  probability  density  function. 
The  probabi?.ity  density  functions  v/ere  determined  by  trial 
and  error:   Knowing  the  mode  and  the  1  percent  end  points,  a 
preliminary  curve  is  dra\m;  the  areas  to  the  left  and  right 
of  th-:  mean  are  determined  and  the  tentative  curve  is  adjusted 
in  such  a  way  as  to  make  these  areas  equal.   If  the  method 
suggef  ted  by  Sobel  and  discussed  in  Chapter  IV  of  this 
study  v-ere  used,  the  cumulative  curve  could  be  sketched 
direct ".y,  since  the  abscissa  values  are  knovm  for  the  1,  10, 
50,  90  and  99  percent  ordinate  values.   This  observation 
prompts  the  recommendation  that  the  Risk  index  methodology 
could  1: 3  made  simpler,  faster,  and  easier  to  use  by  the  ad- 
dition of  a  requirement  for  the  80  percent  central  estimate. 
From  tr;2  cumulative  distributions  of  Figures  16,  17,  18  and 
19,  the  probabilities  of  dollar  and  time  success  (W^   and W  ) 

can  be  determined  for  any  given  values  of  $  and  t 

o      o  * 

SOA-A  Measurement 

The  remaining  input  required  for  the  generation  of  the 
Risk  index  (0.^,)  is  the  state  of  the  art  advance  (Srp)  required 
by  the  project.  The  first  step  in  the  determination  of  S^  is 
a  listing  of  the  critical  subsystems  (X^)  and  assigning  of  the 
relative  weights  (a^.)  .   This  procedure  is  shovm  in  Table  7. 
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Table  7 


DERIVATION  OF  a, 


Subsystem 

Cost 

Relative 
Cost 

Relative 
Importance 

^i 

Airframe 

4/3.4 

.67/1 

.7 

.7/. 8 

Engines 

6/1.6 

1/.47 

1 

1/1 

Avionics 

2/2.2 

.33/. 65 

.5 

.45/. 55 

GSE 

.2/. 52 

.03/. 15 

.1 

.08/. 12 

In  this  example  it  X'zas  not  feasible  to  consider  the 
relative  times  for  the  subsystems;  in  actual  practice  it  V70uld 
be  useful  to  have  this  information.   The  subjective  vreighting 
of  the  relative  importance  and  relative  cost  factors  must  be 
done  on  a  judgment  basis  by  the  analyst.   In  this  case,  it 
was  decided  that  the  relative  importance  V7as  to  be  the  over- 
riding factor;  in  other  cases,  V7here  there  is  less  Spread  in 
the  importance  measurements,  it  may  be  best  to  average  the  two 
factors  or  even  to  weight  the  relative  cost  more  heavily. 

After  the  a^   have  been  generated,  it  becomes  necessary 
to  determine  the  amount  of  increase  in  performance  v^hich  is 
to  be  required  by  each  of  the  subsystems.   The  ratios  of 
required  to  presently  available  capability  (S^) ,  the  estimates 
of  the  percent  of  theoretical  limits  represented  by  each 
subsystem,  the  corresponding  values  of  Tj;  ,  and  the  adjusted 
technological  advance  ratios  (Sj^)  are  displayed  in  Table  8. 
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To  determine  S;i;  is  is  necessary  to  sura  the  products 

RDT&E:        4 

St  =  Z  a^S  •  =  (.065)(.7)  +  (.0585)(1)  +  (.035)(.45) 
i=l 

+  (.018)(.08)  =  .1213 

TOTAL  4 

PROGRAM:   S  =  ^  a  S  '  =  (.065) (.8)  +  (.0585) (1) 
J-   1=1   "■  ^ 

+  (.035)(.55)  +  (.018)(.12)  =  .1320 
Sensitivity  Analysis 

At  this  point  the  only  additional  inputs  required  to 
determine  the  Risk  index  are  the  values  to  be  used  for  $ 
and  t^.   The  unadjusted  probabilities  of  success  for  the  two 
programs  in  terms  of  funding  and  schedule  may  be  determined 
by  referring  to  Figures  16,  17,  18  and  19.   Table  9  summarizes 
the  values  for  the  Risk  and  Success  indices  for  hypothetical 
values  of  $^  and  t^. 

If  it  is  assumed  that  the  firm  will  not  undertake  any 
project  which  has  a  probability  of  success  of  less  than  0.50, 
and  that  the  DOD  has  indicated  the  amount  of  time  which  will 
be  allowed  for  the  completion  of  the  program,  a  distribution 
may  be  generated  which  will  indicate  the  minimum  bid  the 
firm  can  submit  and  the  probability  of  success  which  accom- 
panies any  higher  level  of  funding.   For  example,  consider 
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the  RDT6cE  program  described  above.   If  the  project  is  out- 
lined as  a  five-year  study  contract  and  the  firm  is  asked 
to  bid,  the  distribution  of  Figure  20  could  be  generated  to 
describe  the  chances  of  success. 

Figure  20  shows  that  with  a  60-month  deadline,  the 
minimum  possible  bid  for  a  50  percent  chance  of  success  is 
$1,76  Billion.   Given  70  months  the  bid  could  drop  to  $1.65 
Billion;  if  the  schedule  is  compressed  to  50  months,  the 
minimum  bid  is  $2.62  Billion.   Figure  20  can  be  interpreted 
in  another  way:   given  $^  =  $3  billion,  the  probability  of 
success  for  the  three  values  of  t  can  be  read  directly 
(t^  =  90  months,  frj,   =  .92;  t^  =  60  months,  f     =  .86;  t^  = 
50  months,  j^T  =  .55)  . 

Conclusion 

This  chapter  has  sho\m  the  application  of  the  Risk  index 
derived  in  this  study  to  a  hypothetical  aerospace  program. 
Starting  with  the  performance  specifications,  the  time  and 
budget  were  assumed  and  from  these  the  distributions  for  f 
and]^^  were  generated.   The  specifications  were  used  as  a 
base  for  the  SOA-A  measurement  (S^) ,  and  the  assumption  of 
various  values  for  $^  and  t^  led  to  the  statement  of  the 
corresponding  values  for)/^^.   The  sensitivity  of  ^^  to  changes 
in  $Q  and  t^  was  indicated,  and  a  graphical  technique  was 
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presented  v/hich  permits  analysis  of  the  minimum  bid  figure 
for  a  given  program  length  and  a  stated  probctbility  of 
program  success. 

Exercise  of  the  model  shows  up  several  problem  areas. 
The  first  is  that  the  method  suggested  by  Sobel  would  reduce 
the  effort  required  by  the  analyst  and  perhaps  increase  the 
accuracy  of  the  algorithm.   It  V7as  noted  that  a  significant 
amount  of  judgment  is  involved  in  the  measurement  of  the 
SOA-A.   At  present  it  seerns  unfeasible  as  v/ell  as  undesirable 
to  eliminate  this  subjective  element,  since  lack  of  a  data 
bank  forces  reliance  upon  the  experience  and  expertise  of 
the  individual  analyst.  A  major  weakness  is  the  definition 
of  theTcurve.   The  curve  of  Figure  3  vjas  the  product  of 
extensive  review  of  the  m.eager  store  of  available  data;  lengthy 
conversations  with  experienced  analysts;  thorough  investiga- 
tion of  a  similar  curve  derived  by  Terrell,  and  the  assump- 
tions behind  it;  and  much  thoughtful  consideration  by  the 
author.   It  is  felt  that  until  such  time  as  repeated  use   of 
the  Risk  algoritlira  generates  a  data  bank  of  historical  data 
to  permit  the  definition  of  a  revised  T curve,  the  curve  of 
this  report  represents  as  good  an  estimate  of  the  actual 
relationship  as  can  be  developed. 

In  conclusion,  the  Risk  index  developed  here  is  not 
presented  as  an  ultimate  measure,  but  rather  as  a  tentative 
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step  in  a  direction  V7liich  shov7S  promisee   In  the  final 
analysis,  it  will  be  the  revision  and  modification  of  the 
technique  developed  here  and  above  all  its  usefulness  in 
program  planning  and  project  management  v/hich  \7ill  be  the 
ultimate  measure  of  its  value. 
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